How to Edit Video With an AI Agent: A Free, Open-Source Workflow

By Asim Yousafzai, founder of Con-Gen Media · 8 November 2026

AI agents can now cut silences, write captions and render motion graphics from a plain-English request. They still can't decide what makes your video worth watching. This guide shows the free, open-source stack behind agent-based editing, the order to run it in, and where a human editor still has to step in.

What "editing with an AI agent" means

Most AI video apps give you one button that produces one result. A coding agent such as Claude Code or Codex works differently. It runs command-line tools on your own files, checks the output, and tries again. You describe the edit, and the agent decides which tool to call and in what order.

That matters for three reasons. Your footage can stay on your machine. Every step produces a file you can inspect, such as a transcript or a caption sheet. And when one step is wrong, you can fix that step and rerun it, rather than regenerating the entire video and hoping for a better roll of the dice.

The tradeoff is that you are working with tools, not a polished timeline. You need to be comfortable installing software, and you need to review what comes out.

The free stack, tool by tool

The tools are free, but the agent driving them usually isn't. Claude Code and Codex both run on paid plans or API usage, so check current pricing before you build a routine around one.

One licensing detail catches agencies out. Remotion's free licence covers individuals, non-profits and for-profit organisations with up to three employees. Larger companies need a paid company licence. HyperFrames is Apache 2.0, so it doesn't have that headcount limit. Read each licence yourself before using a tool on client work.

A five-step workflow that holds up

1. Do the first pass on silence

Dead air is the most tedious part of any talking-head edit and the safest part to automate. Auto-Editor analyses audio loudness and removes the quiet stretches:

auto-editor interview.mp4 --margin 0.2s

The margin keeps a little breathing room around each sentence. Without it, speech sounds clipped and rushed. Treat the output as a rough cut, because a loudness check can't tell an awkward pause from a dramatic one.

2. Build the transcript

Run Whisper on the rough cut, not the original, so the timestamps match the footage you are keeping:

whisper interview_ALTERED.mp4 --model small --output_format srt

The transcript is now the most useful document in the project. Your agent can read it to find the strongest opening line, flag repeated takes, or suggest where B-roll belongs. Check names, brand terms and numbers by hand, since those are the words speech recognition most often gets wrong.

3. Let the agent propose the structure

Here is where the agent earns its place. Give it the transcript and a specific brief:

Read this transcript. Find the single sentence that would stop a founder from scrolling. Propose a 30-second cut that opens on it, keep only lines that support it, and list the timestamps.

Ask for timestamps rather than a finished video. A list takes seconds to review, while a bad render wastes minutes. Approve the structure first and render afterwards.

4. Add motion graphics as code

HyperFrames turns HTML, CSS and animation into MP4 files. Its renderer captures each frame in headless Chrome and encodes with FFmpeg, so the same input produces the same video. It needs Node.js 22 or newer plus FFmpeg:

npx hyperframes skills update
npx hyperframes init my-video
npx hyperframes preview
npx hyperframes render

The first command teaches your coding agent how to use the framework. After that you can ask for things in plain language: a lower third with a name and title, a kinetic headline, an animated price comparison. Because a composition is just an HTML file, changing a word or a colour is a small edit instead of a new generation.

5. Burn in captions and export per platform

FFmpeg handles the final assembly:

ffmpeg -i final.mp4 -vf subtitles=interview_ALTERED.srt -c:a copy final_captions.mp4

Before you post, place the frame over a safe-zone guide so captions don't sit under the interface buttons. Our free 1080×1920 safe-zone overlay and export settings sheet are made for that final check.

Where agents still fall short

Taste. An agent can find a sentence that is structurally a good hook. It can't feel that your audience has heard that line a hundred times this month.

Rhythm. Cutting to the beat, holding a reaction shot one moment longer, letting a joke land: these are judgment calls, and they are often the difference between a video people finish and one they abandon.

Sound. Music choice, level balance and well-placed effects still need human ears.

Rights and truth. You remain responsible for the footage, music and claims in the video. If an agent writes on-screen statistics, verify every one of them before it goes live.

The realistic gain is removing the hours of mechanical work so more of your attention goes to these decisions. Used that way, an agent makes a good editor faster. It won't make a weak concept good.

Common mistakes

• Prompting for "a viral video". Vague briefs produce generic edits. Name the audience, the length, the opening line and the single takeaway.
• Rendering before approving. Review the transcript and the shot list first. Text is cheap to change and renders aren't.
• Skipping the watch-through. Watch every export with sound on, start to finish, and once more on a phone.
• Letting the agent overwrite originals. Keep raw footage read-only and have the agent write to a separate output folder.

Frequently asked questions

Can AI edit a video completely on its own?

It can produce a finished file on its own, but quality still depends on human review. Silence removal, transcription and templated graphics automate well. Story, pacing and sound benefit most from an editor's judgment.

Is this workflow really free?

Auto-Editor, Whisper, FFmpeg and HyperFrames cost nothing to use. Coding agents generally require a paid plan or API usage, and Remotion requires a company licence for larger for-profit organisations.

Do I need to know how to code?

Not much, because the agent writes the code. You do need to install the tools, run basic commands and be able to tell when something has gone wrong.

Which tool should I start with?

Start with Auto-Editor on one talking-head clip. It is a single command with an immediately visible result, and it will tell you quickly whether this way of working suits you.