Hundreds of agents at once: waves, sampling and intake

Say you need a short profile for each of four hundred partner shops. No single profile is hard. The number is. A person writing twenty a day needs most of a month. One AI writing them in sequence still takes days, and by the end, the rules from the beginning have long been pushed out of its context.

The obvious move is to run many AIs at once. It is faster, but it brings new problems. Who wrote which profile? Will two AIs edit the same file? Who checks the results? Hand four hundred drafts to one person and that person drowns first.

This post explains how we let hundreds of AIs work at the same time while a person still reads only one card. The point is not how capable the AI is. It is deciding what goes to AI and what we insist on giving to plain programs.

AI does one item

The most important rule in the whole design: each AI is responsible for exactly one item. Splitting, queuing, checking, collecting and record keeping need no judgment, so programs do them. A program gives the same result every run and uses none of the AI budget.

There are two reasons for this split. First, the smaller the part that needs judgment, the fewer mistakes AI makes. Writing one profile is far easier than managing progress on four hundred. Second, repeatable steps belong to programs so that, when something breaks, the cause can be found. If AI did the queuing and counting too, each run might differ, and a failure would leave no trail.

We do not start at hundreds, either. The pipeline is worth setting up only when similar work piles up — dozens of items or more. For a handful, one AI working through them in order is simpler.

Work orders do not change

The pipeline starts from two things a person has confirmed: a spec that says what each profile should look like, and a target list of every shop. A program combines them into one work order per item.

Once created, a work order is frozen. A worker cannot adjust the topic or scope because it thinks something else would be better. If the spec really is wrong, the right move is to stop, fix the spec and regenerate the orders, not to let four hundred AIs each make their own interpretation.

Each work order also names its own output folder. A worker may create or change files only inside that folder. It cannot touch anyone else's, and it may not run any version-control command. One item, one folder, one writer: two AIs editing the same thing becomes impossible by construction.

Waves, one at a time

The four hundred orders are not released at once. They are cut into waves, for example a hundred items each. The dispatcher handles one wave at a time and caps how many run together — say six at once — while the rest wait in a queue. The cap depends on the machine and the budget.

Waves do three things. Mistakes surface in the first wave; if the spec has a hole, at most one wave is wasted, not all of them. People get a rhythm: at the end of each wave, someone reviews the results, makes a call, and only then is the next wave released. And cost stays under control. Before the real run we try about ten items, measure time and budget per item, multiply by the total, and release the first wave only if it fits.

An example. After the first wave, collection shows a dozen items stuck in the same place: the spec asks for opening hours, but one kind of shop opens only on festival days and has no fixed hours. That is not the workers' fault; the spec never considered the case. A person sees it on the card, adds one line to the spec, and regenerates the orders for the remaining waves. The next three hundred items avoid the same trap. Had all four hundred gone out at once, that dozen would have been fifty or sixty.

The dispatcher itself does not commit anything or change any status. It hands out work orders, collects results, and records who delivered, who got stuck, who failed and who went silent. Committing and status updates wait until after collection, when the coordinating session handles them in one step after a person signs off.

Self-reports are not evidence

Before handing in, a worker runs the checker on its own output: the file exists, the length is in range, required fields are there, there are enough cited sources, no placeholders such as "TBD" remain, and there are no stray files. Then it reports, "I checked; everything passes."

We do not take that at its word. Not because AI lies, but because a model's description of its own work is not reliable evidence. It may have checked the wrong file, misread the result, or never finished the run. So during collection, a program re-runs the very same checker on every item. Only that result counts.

The checker itself has to be trustworthy too. It is an ordinary program under version control, with its own tests, and changing it needs a person's sign-off. Workers have no permission to edit it, so none of them can loosen the standard to get its own work through.

Using the same checker matters. If workers used one standard and collection used another, the two could disagree and no one could say which was right. There is one standard. Workers self-check to catch problems early; collection re-checks so people can trust the outcome.

"I checked" is a sentence, not evidence. Evidence is a check that anyone can re-run and get the same result.

Sampling looks for errors, never edits

Mechanical checks confirm format, not truth. A profile can be perfectly formatted and still list the wrong opening hours. No checker will notice. That needs a second pair of eyes, but asking a person to read all four hundred is too much.

Instead, a fixed share is sampled — one in twenty, for example — and sent to a separate AI whose only job is review. Its task is narrow: pick a few factual claims from the output and try to find evidence against them. It does not polish the writing and it cannot change the original. It only writes review notes, stored separately.

The reviewer is always a different AI from the author, and it never sees the author's self-report — only the output and the original sources. That is deliberate. A reviewer that first reads "I checked; it is all correct" tends to follow the author's line of thought instead of looking on its own.

Why not let the reviewer just fix things? Because once it edits, nobody can tell whose judgment the final version reflects, and a mistake can no longer be traced. Notes stay separate and the author reads them on a retry. The original stays as it was, and the record stays whole.

Sampling results are also a signal. If almost every sampled item is fine, accepting the wave is a reasonable call. If a high share fails review, the problem is likely not a few items but the spec or the work orders. Then the whole wave pauses for a person, instead of letting AI keep redoing it.

Four statuses, computed by a program

At any moment, every item is in exactly one of four states: not started, retrying, passed, or needs a decision. Workers do not declare these. The collector reads the outputs and check results, computes the status, and appends it as an event.

The rules are simple. All checks pass and, if sampled, the review passes: passed. A check fails and there has been no retry yet: it goes on the retry list. It fails again after one retry, or that wave's review failure rate is high: needs a decision. Items not yet reached are not started.

There is only one retry. A failed item goes back to the dispatcher with its reason — "only two sources cited," or "opening hours do not match the source" — so the next worker knows what went wrong. If the second attempt fails as well, a third AI attempt rarely helps. That is the moment for a person.

Each event records which item, which worker or program wrote it, when, what status was assigned, and which check result it rests on. Events are only ever appended. If a verdict later turns out wrong, a correcting event is added; the original is never edited. Any item's full history can then be read end to end: which wave it joined, why it first failed, who reviewed the retry, and on which card a person accepted it.

Because every status lives in an append-only ledger, the progress tree and the decision card are generated from it, never written by hand. When a person sees "380 passed, 12 need a decision," every number traces back to specific events.

Intake, and accepting a wave

Even items that pass every check do not go straight into the official record. They go into an intake area first. The intake area has one property: people and AI can put things in, but nothing there can change the record directly.

What a person reviews here is one card, not four hundred drafts. It usually has three to seven items: how many passed, which need a decision, what kinds of problems review found, and a suggested way forward. The person replies in a single line, such as "1 yes, 2 hold," and the system writes the accepted items into the record as one wave, leaving held items in intake.

Held or returned items do not vanish. They stay in intake with the reason attached. They can be fixed and resubmitted in a later wave, or a person can decide to drop them. Either way, the outcome is written as an event and can be found later.

The design comes from a practical observation. In large batches, the bottleneck is often not AI but human approval. If a person must approve each item one by one, the approval queue grows and grows until nothing gets through. Accept in bulk whatever passes the mechanical checks, show people only the exceptions, and their attention goes where judgment is actually needed.

So what do people actually do in this flow? Look at the whole thing and they appear in just three places: confirming the spec and target list at the start, reading one card and replying in one line between waves, and making the final call on items that need a decision. The rest of the time, nobody has to watch.

That also means each human decision carries weight. A person confirmed the spec, so four hundred AIs pull in one direction. A person accepted each wave, so every entry in the record has someone accountable for it. AI handles the volume. People handle the judgment.

A last word on limits. Each program in this pipeline can be tested on a few examples, but only a full run of hundreds shows the problems that appear at scale — a budget running out early, or one kind of shop whose details are unusually hard to find. So we treat the first wave as a trial and decide on the next one after seeing it, rather than releasing everything at once.

In short

• AI does one item; splitting, queuing, checking and collecting are programs that behave the same every time.
• Work orders are frozen, and each item has its own folder and a single writer, so nothing overwrites anything else.
• Work runs in capped waves, after a small trial run, so mistakes show up in the first wave.
• A self-report is not evidence: one checker re-runs on every item, and sampled review only looks for errors.
• Results wait in intake; a person reads one card of exceptions and accepts or returns a whole wave.