November 13, 2026 · 10 min read
My video studio is code. A new benchmark tested agents on the same job.
AgenticStudio is my name for the code that makes my YouTube channel. It makes four things:
- Shorts most days.
- A Sunday digest.
- A cartoon debate show called Pick a Side.
- A kit for the videos I present on camera myself.
Not one frame comes from an image or video model. Every slide, chart, cartoon owl and subtitle is drawn by code, and the only generated media is the voice. The photos are real ones, from the articles or a stock library.
Last month a paper called VibeVideoBench tested frontier AI agents on exactly this kind of work. Reading it next to my own repo was more useful than most benchmark papers, because it measures the problem I actually have. This post is a tour of the studio, held up against that paper.
What VibeVideoBench measures
The authors start from a point I agree with completely. Diffusion video models make striking clips, but they cannot reliably render an exact formula, a correct chart or narration that lines up with what is on screen. The other route is an agent that writes code to render every frame and synthesize every sound.
The setup:
- 239 tasks in 12 categories, in English and Chinese, from data stories and music-driven pieces to classroom lessons and paper explainers.
- Each agent runs in its own vendor harness, inside a sandbox with no image or video model, and has one hour to deliver a finished MP4 with sound.
- Grading in three levels: program checks of the delivery spec, then about 13,000 task-specific rubric checks, each traced to the requirement it enforces, then five quality dimensions scored by three panels of vision-language model judges.
Across nine agent systems and 2,151 scored runs, GPT-6 Astra and Claude Opus 5.5 came first and second, for the judges and for blind human raters alike. The best system met every core requirement on 76.3% of tasks.
Two findings matter more than the ranking:
- Missing videos change the ranking. About three quarters of the lost videos were final renders that started too late, ran too slowly or never started. The authors call it a failure of how agents plan their hour.
- Agents inspect their own frames, but how much they look does not predict quality.
Both match what I have learned the slow way.
What AgenticStudio makes
- Shorts. An hourly job reads my blog and a set of tech news feeds. A model judges each story: make it, rewrite the angle, or skip it. A Short is about a minute of narrated, animated slides in the channel’s look, and it goes public only after a final check against the article.
- The Sunday digest. It’s 20 to 30 minutes long and retells the two to four stories whose Shorts did best in their first 72 hours. It builds on Sunday morning and waits for me in a dashboard. Nothing uploads until I click Publish.
- Pick a Side. I fill in a form: the question, two to four panelists and the side each one argues, plus a few source links. The pipeline researches the topic into a ledger of quoted facts, writes the script and renders a cartoon studio with animal panelists behind desks and an owl as host. The show never gives a verdict. It ends with “Which side convinced you? Tell us in the comments.”
- The presenter kit. This is the human version. It turns a topic into a research brief, a teleprompter script and slides with a reserved band at the bottom for my webcam. After I record, it aligns the known script to my voice with WhisperX and writes English subtitles, then German, French and Persian translations. I built it after the faceless videos lost their novelty: viewers stopped clicking after the first week.
Every format follows the same shape. The models fill in steps and code checks them. A Short goes out on its own only if every check passes; the digest and the show always wait for my click.
The render side is plain: Pillow draws the slides, and ffmpeg animates matted layers and mixes the sound. I picked boring tools because they are deterministic. The same script renders the same video, so a bug can be reproduced and a fix can be tested.
The voices are Google’s Chirp 3 HD TTS. They have quirks no benchmark would list: Chirp spells “Hmph” and “Pfft” letter by letter, so the panelists scoff with “Oh, please” and “Yeah, right” instead. A real laugh comes from a second TTS model that understands a [laughing] tag.
All of it runs on the same always-on mini-PC as my agent harness. Over 30 days AgenticTube, the automatic part, was about half of the 3.66 USD my apps spent on models. Text-to-speech is billed separately.
Where the paper and the studio agree: code draws exact things
The paper’s opening argument has been the studio’s first rule since version one: no AI imagery and no invented numbers. A chart is drawn from values that came out of the source, so a chart cannot hallucinate.
Sync is the same story. Chirp takes no timing marks, so I get no word timestamps. Each card’s moment is computed instead: where its words first appear in the text, as a share of the text, times the length of the audio. It is accurate to about half a second, which is enough for a card to arrive as the narrator says it. A diffusion model would have to guess that.
Where they differ, part 1: my agents do not plan their hour
The paper’s agents build a rendering pipeline from scratch inside a one-hour budget, and most of the videos they lose die at the render. Mine never face that decision, because the pipeline is fixed and its costs are known:
- A Short renders in about 22 seconds.
- A 12-minute debate episode takes about two minutes on six workers.
- A test build of a 26-minute digest took 20.5 minutes wall clock and 2 hours 24 minutes of CPU, against a 3-hour limit on the job.
I only know those numbers because I rehearse: a copy of the live database in a sandbox, a fake YouTube client, and the real job’s own settings. In my studio the model writes the words. Scheduling, retries and the render are plain code.
That is the same conclusion I reached about general agents versus fixed pipelines, now with a benchmark behind it. When the job is known, asking a model to plan it on every run adds a failure mode that buys nothing.
Where they differ, part 2: rubric checks or gates
VibeVideoBench grades every video against about 54 rubric checks on average, each traced to a requirement. That is the right idea, and I do the same thing, except my checks run before publishing and block it:
- Every number on screen or on air must match a cited fact by value and unit. That includes spelled-out numbers, “two thirds of”, “$5bn” and “10x”.
- At most three figures per Short, because the model’s brief pulls out every number in the article, and the early scripts read like earnings calls.
- Every Short ends with a neutral question.
- The debate never gives a verdict or advice.
- A panelist may cite a fact in at most two lines. No sentence may share 60% or more of its content words with one that panelist already said.
None of these came from a spec. Each came from watching an episode. The repetition rule exists because one episode cited the same fact in 13 lines from the same panelist. A rubric tells you afterwards that a video failed. A gate tells the pipeline to try again before anyone sees it.
Where they differ, part 3: looking is not quality
The paper’s finding that how much agents look does not predict quality is the one I would underline. My own version: pixel tests passed four real visual defects. The tests checked that each frame rendered, had content and fitted its layout. None of them noticed a photo squeezed to a thumbnail, or a headline cut off with room to spare.
What caught them was a contact sheet of real renders that I looked at after every change. Then there are the things no frame check knows about. YouTube draws the channel name, the title and the Subscribe button over the bottom of a Short, and the published figures for that safe zone disagree with each other. My content now stays above row 1520 of 1920 and 140 pixels from both edges, and the final check was holding the phone in my hand.
What no benchmark measures: whether anyone watches
Here is the honest part. For a while, Shorts on heavy topics got 2 to 37 views, and 0 to 7% of each one was watched. A friend outside tech told me the explanations were hard to follow. Every one of those Shorts would have passed a strict rubric.
So I retargeted the channel at that friend:
- General news feeds instead of most vendor engineering blogs.
- One Short per event.
- Claims wait for an on-record statement or a second independent outlet; opinions are skipped.
- At most three numbers, and no unexplained jargon.
That change was a decision about an audience, not a fix to a pipeline. A benchmark can say whether an agent met 76% of a spec. It cannot tell you whether the spec was worth meeting. That is why the long formats still end with me clicking Publish or Rework, and why a Short that fails a check waits for me too.
Reading the paper with care
- Each agent ran in its own vendor harness, so the ranking scores a model and its harness together. A good model in a weak harness and a weak model in a good one cannot be told apart.
- The judges’ ranking matched blind human preferences. That is a statement about ranking systems, not proof that a judge can approve a single video. The Publish button stays.
- I have read the abstract and a third-party review, not the full paper. The numbers above come from the abstract.
What I am taking from it
The paper gives a name to something I had only felt: delivery is its own axis. A video that is 90% good and arrives is worth more than a perfect one that never finishes rendering. Fixed pipelines win that axis by default, and that is most of why AgenticStudio looks the way it does.
The part I want to steal is the traceability. Every rubric check in VibeVideoBench points to the requirement it enforces. My gates do not yet say which of my decisions they come from, and after five months of rules that started with “I noticed”, they should.