video-use — Agent 剪辑师 / Agent-Directed Video Editor

Wraps the open-source browser-use/video-use project (MIT). Instead of preset-driven editing, the agent reasons from the transcript + on-demand visuals. Transcription runs through the published Clawvard SDK; everything else runs locally on ffmpeg. Zero third-party API key, one Clawvard API key.

Upstream credit: the editing pipeline, Hard Rules and self-eval design come from browser-use/video-use (MIT). This skill only overlays the transcription helper — the rest is upstream unchanged. The overlay itself ships as a public npm package (clawvard-video-use-adapter); users never touch a private repo.

Principle

1. LLM reasons from raw transcript + on-demand visuals. The only derived artifact that earns its keep is a packed phrase-level transcript (takes_packed.md). Everything else — filler tagging, retake detection, shot classification, emphasis scoring — you derive at decision time.
2. Audio is primary, visuals follow. Cut candidates come from speech boundaries and silence gaps. Drill into visuals only at decision points.
3. Ask → confirm → execute → iterate → persist. Never touch the cut until the user has confirmed the strategy in plain English.
4. Generalize. Do not assume what kind of video this is. Look at the material, ask the user, then edit.
5. Artistic freedom is the default. Every specific value, preset, font, color, duration, and technique in this document is a worked example, not a mandate. The only things you MUST do are in the Hard Rules section below. Everything else is yours.
6. Verify your own output before showing it to the user. If you wouldn't ship it, don't present it.

Hard Rules (production correctness — non-negotiable)

These are the things where deviation produces silent failures or broken output. Not taste, correctness.

1. Subtitles are applied LAST in the filter chain, after every overlay. Otherwise overlays hide captions.
2. Per-segment extract → lossless -c copy concat, not single-pass filtergraph. Otherwise you double-encode every segment.
3. 30 ms audio fades at every segment boundary (afade=t=in:st=0:d=0.03,afade=t=out:st={dur-0.03}:d=0.03). Otherwise audible pops at every cut.
4. Overlays use setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start.
5. Master SRT uses output-timeline offsets: output_time = word.start - segment_start + segment_offset.
6. Never cut inside a word. Snap every cut edge to a word boundary from the transcript.
7. Pad every cut edge. Working window: 30–200 ms. cv.media.transcribe returns segment-level timestamps; the adapter's align_words.py rewrites them to word-level via ffmpeg silencedetect + faster-whisper, so cuts remain word-accurate.
8. Word-level verbatim ASR only. The adapter emits the same words[] schema upstream helpers expect — never fall back to phrase-only.
9. Cache transcripts per source. Never re-transcribe unless the source file itself changed. edit/transcripts/<name>.json is the cache.
10. Parallel sub-agents for multiple animations. Never sequential. Spawn N at once via the Agent tool.
11. Strategy confirmation before execution. Never touch the cut until the user has approved the plain-English plan.
12. All session outputs in <videos_dir>/edit/. Never write inside the video-use/ project directory.

Transcription — two paths, one schema

helpers/transcribe.py supports two transcription paths and picks between them automatically. Both produce the same JSON payload (segments[] + words[]), so every downstream helper (pack_transcripts.py, render.py, master.srt builder) is path-agnostic.

Path A · Clawvard SDK (preferred). cv.media.transcribe (Volcengine Doubao 2.0) returns segment-level utterances — best-in-class accuracy on Chinese + English podcast material. Because the SDK output is segment-level only, the adapter rewrites it into word-level before caching:

1. Silence-snap edges. helpers/align_words.py runs ffmpeg -af silencedetect=n=-35dB:d=0.15 on the extracted mono/16 kHz audio, gathers every silence span, and snaps each segment's start/end to the nearest silence-gap midpoint within a 200 ms window. Absorbs the drift the segment API returns.
2. Word timestamps via faster-whisper. faster-whisper (MIT, offline, no key) runs over the same audio with word_timestamps=True. The resulting words are merged into the payload in the upstream-compatible schema:

   {"type": "word",    "text": "…", "start": 0.83, "end": 1.02}
   {"type": "spacing", "text": " ", "start": 1.02, "end": 1.31}
   

Path B · Local faster-whisper (fallback). A single faster-whisper pass produces segments[] and words[] in one go. Zero SDK calls, zero third-party key, zero Clawvard credits — everything runs on CPU with int8 quantization. Slightly less accurate on tricky audio than Doubao 2.0, but strictly good enough for the whole video-use pipeline. Both paths run directly against their native SDK / library; neither goes through an intermediate proxy or a paid third-party ASR service.

Mode selection (--mode on transcribe.py, or VIDEO_USE_TRANSCRIBE env var):

• auto (default) — try SDK first; on any SDK error (missing key, quota, transient server config, network) fall back to local with a warning printed on stderr. This is the mode popularTask #1 uses; it means the course finishes end-to-end even when the SDK is unavailable.
• sdk — force SDK; fail loudly if the Clawvard API key is missing or the call errors. Use when you want the higher Doubao accuracy and want to be surfaced any config issue immediately.
• local — skip the SDK entirely. Use for a fully offline run (e.g. sensitive material that must not leave the machine).

Model choice: base.en (74 MB) for English-only, small (466 MB) for multilingual. Override with WHISPER_MODEL=…. First run downloads the model into ~/.cache/huggingface/; every subsequent run is offline.

If faster-whisper produces an empty word list for a segment (rare, e.g. all-silence input), the segment is emitted with words: [] and the agent MUST warn the user rather than pretend the cut was word-accurate. Do not silently ignore.

Directory layout

The skill lives in video-use/. User footage lives wherever they put it. All session outputs go into <videos_dir>/edit/.

<videos_dir>/
├── <source files, untouched>
└── edit/
    ├── project.md               ← memory; appended every session
    ├── takes_packed.md          ← phrase-level transcripts, the LLM's primary reading view
    ├── edl.json                 ← cut decisions
    ├── transcripts/<name>.json  ← cached transcribe.py output (segments + word-level align)
    ├── animations/slot_<id>/    ← per-animation source + render + reasoning
    ├── clips_graded/            ← per-segment extracts with grade + fades
    ├── master.srt               ← output-timeline subtitles
    ├── downloads/               ← yt-dlp outputs
    ├── verify/                  ← debug frames / timeline PNGs
    ├── preview.mp4
    └── final.mp4

Setup

First-time install lives in install.md (clone upstream, overlay adapter, deps, ffmpeg, SDK, skill registration, API key). Don't re-run it every session; on cold start just verify:

• The Clawvard API key resolves — either in the environment or in .env at the video-use repo root. If missing, ask the user to paste one and write it to .env (never to the user's <videos_dir>).

• ffmpeg + ffprobe on PATH.

• Python deps installed (uv sync or pip install -e . inside the repo) + faster-whisper.

• Node 18+ and the Clawvard SDK reachable. Verify with:

  node -e "import('@clawvard/sdk').then(m => console.log(!!m.Clawvard))"
  # → true
  

• Node.js + npm available if the session needs HyperFrames or Remotion slots. HyperFrames currently requires Node.js 22+.

• yt-dlp, HyperFrames, Remotion, Manim installed only on first use.

Helpers (helpers/transcribe.py, helpers/align_words.py, helpers/render.py, etc.) live alongside this SKILL.md. Resolve their paths relative to the directory containing this file.

Helpers

• transcribe.py <video> — single-file transcription: extract audio → cv.media.transcribe → silence-snap + faster-whisper word alignment → cached JSON. Cached.
• transcribe_batch.py <videos_dir> — 4-worker parallel transcription of every video in the folder. Same cache policy.
• align_words.py — the segment→word rewriter; called from transcribe.py, not usually run standalone.
• pack_transcripts.py --edit-dir <dir> — transcripts/*.json → takes_packed.md (phrase-level, break on silence ≥ 0.5 s).
• timeline_view.py <video> <start> <end> — filmstrip + waveform PNG. On-demand visual drill-down. Not a scan tool — use it at decision points, not constantly.
• render.py <edl.json> -o <out> — per-segment extract → concat → overlays (PTS-shifted) → subtitles LAST. --preview for 720p fast. --build-subtitles to generate master.srt inline.
• grade.py <in> -o <out> — ffmpeg filter-chain grade. Presets + --filter '<raw>' for custom.

For animations, create <edit>/animations/slot_<id>/ with Bash and spawn a sub-agent via the Agent tool.

The process

1. Inventory. ffprobe every source. transcribe_batch.py on the directory. pack_transcripts.py to produce takes_packed.md. Sample one or two timeline_views for a visual first impression.

2. Pre-scan for problems. One pass over takes_packed.md to note verbal slips, obvious mis-speaks, or phrasings to avoid. Plain list, feed into the editor brief.

3. Converse. Describe what you see in plain English. Ask questions shaped by the material. Collect: content type, target length/aspect, aesthetic/brand direction, pacing feel, must-preserve moments, must-cut moments, animation and grade preferences, subtitle needs.

4. Propose strategy. 4–8 sentences: shape, take choices, cut direction, animation plan, grade direction, subtitle style, length estimate. Wait for confirmation.

5. Execute. Produce edl.json via the editor sub-agent brief. Drill into timeline_view at ambiguous moments. Build animations in parallel sub-agents. Apply grade per-segment. Compose via render.py.

6. Preview. render.py --preview.

7. Self-eval (before showing the user). Run timeline_view on the rendered output at every cut boundary (±1.5 s window). Check each image for:

   • Visual discontinuity / flash / jump at the cut
   • Waveform spike at the boundary (audio pop that slipped past the 30 ms fade)
   • Subtitle hidden behind an overlay (Rule 1 violation)
   • Overlay misaligned or showing wrong frames (Rule 4 violation)

   Also sample: first 2 s, last 2 s, and 2–3 mid-points — check grade consistency, subtitle readability, overall coherence. Run ffprobe on the output to verify duration matches the EDL expectation.

   If anything fails: fix → re-render → re-eval. Cap at 3 self-eval passes — if issues remain after 3, flag them to the user rather than looping forever. Only present the preview once the self-eval passes.

8. Iterate + persist. Natural-language feedback, re-plan, re-render. Never re-transcribe. Final render on confirmation. Append to project.md.

Cut craft (techniques)

• Audio-first. Candidate cuts from word boundaries and silence gaps.
• Preserve peaks. Laughs, punchlines, emphasis beats. Extend past punchlines to include reactions — the laugh IS the beat.
• Speaker handoffs benefit from air between utterances. Common values: 400–600 ms. Less for fast-paced, more for cinematic. Taste call.
• Silence gaps are cut candidates. Silences ≥ 400 ms are usually the cleanest. 150–400 ms phrase boundaries are usable with a visual check. < 150 ms is unsafe (mid-phrase).
• Example cut padding: 50 ms before the first kept word, 80 ms after the last. Tighter for montage energy, looser for documentary. Stay in the 30–200 ms working window (Hard Rule 7).
• Never reason audio and video independently. Every cut must work on both tracks.

The packed transcript (primary reading view)

pack_transcripts.py reads all transcripts/*.json and produces one markdown file where each take is a list of phrase-level lines, each prefixed with its [start-end] time range. Phrases break on any silence ≥ 0.5 s. This is the artifact the editor sub-agent reads to pick cuts — it gives word-boundary precision from text alone at ~1/10 the tokens of raw JSON.

Example line:

## C0103  (duration: 43.0s, 8 phrases)
  [002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
  [006.08-006.74] S0 We fixed this.

Subtitles (when requested)

Subtitles have three dimensions worth reasoning about: chunking (1/2/3/sentence per line), case (UPPER/Title/Natural), and placement (margin from bottom). The right combo depends on content.

Worked style — pick, adapt, or invent:

bold-overlay — short-form tech launch, fast-paced social. 2-word chunks, UPPERCASE, break on punctuation, Helvetica 18 Bold, white-on-outline, MarginV=35. render.py ships with this as SUB_FORCE_STYLE.

FontName=Helvetica,FontSize=18,Bold=1,
PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,BackColour=&H00000000,
BorderStyle=1,Outline=2,Shadow=0,
Alignment=2,MarginV=35

Invent a second style (narrative, documentary, education — 4–7 word chunks, sentence case, MarginV=60–80) if that fits better. Hard rules: subtitles LAST (Rule 1), output-timeline offsets (Rule 5).

Output spec

Match the source unless the user asked for something specific. Common targets: 1920×1080@24 cinematic, 1920×1080@30 screen content, 1080×1920@30 vertical social, 3840×2160@24 4K cinema, 1080×1080@30 square. render.py defaults the scale to 1080p from any source; pass --filter or edit the extract command for other targets. Worth asking the user which delivery format matters.

EDL format

{
  "version": 1,
  "sources": {"C0103": "/abs/path/C0103.MP4", "C0108": "/abs/path/C0108.MP4"},
  "ranges": [
    {"source": "C0103", "start": 2.42, "end": 6.85,
     "beat": "HOOK", "quote": "...", "reason": "Cleanest delivery."},
    {"source": "C0108", "start": 14.30, "end": 28.90,
     "beat": "SOLUTION", "quote": "...", "reason": "Only take without the false start."}
  ],
  "grade": "warm_cinematic",
  "overlays": [
    {"file": "edit/animations/slot_1/render.mp4", "start_in_output": 0.0, "duration": 5.0}
  ],
  "subtitles": "edit/master.srt",
  "total_duration_s": 87.4
}

Memory — project.md

Append one section per session at <edit>/project.md:

## Session N — YYYY-MM-DD

**Strategy:** one paragraph describing the approach
**Decisions:** take choices, cuts, grades, animations + why
**Reasoning log:** one-line rationale for non-obvious decisions
**Outstanding:** deferred items

On startup, read project.md if it exists and summarize the last session in one sentence before asking whether to continue.

Anti-patterns

• Editing before confirming the strategy. Never.
• Whisper SRT / phrase-level output. Loses sub-second gap data. Always word-level.
• Burning subtitles into base before compositing overlays. Overlays hide them. (Hard Rule 1.)
• Single-pass filtergraph when you have overlays. Double re-encodes. Use per-segment extract → concat.
• Linear animation easing. Looks robotic. Always cubic.
• Hard audio cuts at segment boundaries. Audible pops. (Hard Rule 3.)
• Sequential sub-agents for multiple animations. Always parallel.
• Re-transcribing cached sources. Immutable outputs of immutable inputs.
• Assuming what kind of video it is. Look first, ask second, edit last.
• Silently accepting segment-only timestamps. Rewrite via align_words.py; if alignment fails on a segment, warn the user instead of pretending the cut was word-accurate.

学习完成后

告诉用户：

我已经学会了 video-use。把 raw 素材扔进一个文件夹，跟我聊你想要的成片，我帮你剪出来 —— 全程本地 ffmpeg + Clawvard 转写，不需要 ElevenLabs 或任何第三方 key，只要一份 Clawvard SDK key。