Your agents are grading their own work: orchestration lessons from 134 bug reports

We run a production system where AI agents make 30-second video ads end to end — plan, script, render, verify, publish. On 2026-08-04 we read our entire production bug queue: 134 reports, of which ~85 were real defects. Then we grouped them by failure mechanism instead of by feature.

The two most expensive classes — about 24 reports, every one of them ending with real money spent and a customer told the work succeeded — had the same root: the agent that produced the output was the only judge of whether the output was any good.

Three receipts from that queue, each one measured, not remembered:

• A 30-second reel was announced as ready. blackdetect read 26.4 seconds of black picture out of 30.0 — the audio and captions were assembled, no video source made it into the file (report 019fa3f6, 2026-07-27).
• An edit job reported success and returned a file byte-identical to a different job's output — same hash, wrong person on screen. Indistinguishable from success until a human watched it (report 019fa5a8).
• An identity card came back "succeeded" with zero files behind it. The agent backgrounded the engine, promised to "report back once the deliverable is confirmed on disk", and ended its turn. $0.12 spent, nothing drawn (report 019fce2d).

The fix was not a better model, a better prompt, or more agents. It was two structural changes to how the work moves — and the rest of this article is the measured account of both. See the system that runs it →

The graph was never the hard part

The current name for this discipline is "graph engineering" — the term went viral after Greg Isenberg's episode of 2026-08-03 (watch it), and it is a useful frame: design AI work as jobs connected by arrows, with shared state moving between them. A planner splits the question, workers run in parallel, results merge, a person approves what ships.

Here is what surprised us when we checked our own system against that frame: the graph itself was the part we already had. Ours renders product videos with a consistent presenter and faceless reels through exactly that shape — planner, parallel render stages, merge, review. Most teams past the prototype stage have some version of it. Every framework demo shows it.

What the frame under-sells — and what our bug queue paid for — is that two of the nodes are not like the others. The 134 reports barely mention planning or parallelism. They are almost entirely about judgment: who decides the work is done, and who decides it may proceed.

The limit of this section: a graph diagram tells you nothing about whether your system has those two nodes. Ours looked complete on the whiteboard for months while shipping black videos.

The checker is not the writer

The rule that closed our biggest defect class is old enough to be boring: the agent that composed the output does not get the final word on whether the output is deliverable.

What made it work in practice was making the checker structural instead of per-bug. After each of the three failures above we had shipped a point fix; the class kept coming back on the next kind of output. What stopped it (shipped 2026-08-08) was a postcondition registry: every render kind in the system must declare its delivery checks — file exists, non-empty, playable, the ordered slide count at the ordered pixel size, a readable image — and the pipeline runs them after the engine reports success and before the job may say "succeeded". A test pins the registry against the list of kinds, so a new kind cannot ship without naming what "delivered" means.

Two design choices that mattered more than we expected:

• Deterministic checks first, model judgment last. Almost everything worth checking is measurable with ffprobe and a hash: duration, pixel size, black frames, "the output is not a byte-copy of an input". We reserve a vision-model check for the one thing measurement cannot see — is this the same face and the same product label across frames — and it runs advisory until its false-positive rate on real renders is known.
• A failing check is a failed job with a plain-language reason — never a delivered defect with a warning attached. An honest failure costs less trust than a black rectangle announced as "ready".

The best evidence that the rule generalises: it caught our own checker. The first version counted a carousel's slides by image files only; the reviewer agent on our push gate — a checker that is not the writer, pointed at code — flagged that animated slides ship as .mp4, so every animated deck the engine had built exactly as ordered would have been failed as defective. One blocked push instead of a production incident.

The limit: a delivery check can only judge what is on disk. It cannot tell you the video is boring — that stays with the human gate below, and with the format checks that run before money is spent, not after.

The gate is a pause, not two jobs

Our human approval gates used to be surgery. The pipeline that replicates a competitor's ad format needs a person to approve the product card and starting frames before the paid render — so we had cut it into two separate jobs with a second submit and a second estimate, because a job could not wait.

The orchestration fix (shipped 2026-08-08) was to make waiting a job state. An engine that reaches a checkpoint writes a small gate-request file — what is being asked, in plain words, and which artifacts to judge — and the job parks as waiting_approval. Approval flips it back into the queue in one guarded database statement (a double-click or a race is a refusal, never a second transition), the parked artifacts are staged back, and the same job continues. Two submits became one; the second estimate disappeared; the hand-off bridge between the halves stopped being a failure surface.

Two production lessons came with it, both caught by review before they shipped:

• A parked job nobody hears about is parked forever, billed. Our first cut wrote the notification and then filtered it out of both the feed and the push list. If your orchestrator has a human-in-the-loop state, the loudest surface you own — a phone push, in our case — belongs to "something needs your decision".
• The money must be bounded before the gate, not after. The single job's estimate prices the paid half as an upper bound (per-beat seconds clamped to the same constant the engine clamps to), so the ceiling the customer approved covers exactly what can be spent. The gate message then shows the measured plan — beats, seconds, anything dropped — before a cent of the render runs.

The limit: a gate is only as good as the question it asks. Ours sends the measured plan and the artifacts; a gate that asks "approve?" with no evidence is a rubber stamp with extra latency.

What the runs cost

Customer prices as quoted by the system's own estimates, dated 2026-08-08; the render figures are from the published run of 2026-08-05.

The limit: these are our catalogue's prices on the dates named — models reprice, and the estimate before each run is the number that binds.

The mistakes, each with its price

1. Same model writes and grades. Price: a paid 30-second render delivered as 26.4 seconds of black, found by the customer (report 019fa3f6). This is the default behaviour of every single-agent loop.
2. Point fixes for a structural class. Price: three fixed bugs and the same class back on the fourth kind of output — until the postcondition registry made "what does delivered mean" a required declaration per kind.
3. The checker inherits the writer's blind spot. Price: one blocked push. Our image-only slide count would have failed every valid animated carousel. Review your checkers like code, with a checker that did not write them.
4. A human-in-the-loop state without a reachable human. Price: two blocking review findings. The notification existed; the feed and the push allow-lists filtered it. Every parked job would have waited forever with the prep money already spent.

An ai agent orchestration checklist you can steal

Distilled from the epic that closed the class — six items, each one paid for above:

1. Draw the smallest graph that improves quality. More agents means more noise; we added zero agents and two node types.
2. Every producing node names its postconditions, and a checker that is not the writer runs them before "done" may be said.
3. Deterministic checks before model judgment; a model-judge runs advisory until you have measured its false-positive rate.
4. A failing check fails the job with a reason a person can read.
5. Waiting for a human is a first-class state — reapers must not kill it, the loudest notification surface you own must announce it, and approval must be race-safe.
6. Bound the money before the gate; show the measured plan at it.

None of this needs a framework to start. Ours is plain queue rows and one registry file; the influencer pipeline ran the manual version of the same graph long before the states had names.