The AI learning loop: a three-time threshold and human sign-off

Picture a small design studio. Whenever a course project comes in, an AI assistant fills in a starting template: goals, how success is measured, what gets delivered. The first time, the project lead changes the handout from Chinese only to bilingual. The second time, the same edit. The third time, the same edit again.

At that point any good assistant would speak up: "Should course handouts just be bilingual by default from now on?" This article is about teaching an AI to ask that question at the right moment. Not after the first edit, when it would be guessing. Not after the tenth, when everyone is tired of making it.

We call this the learning loop. At its heart are two things: a count of three, and a person's sign-off.

Why not let the AI learn on the spot

Having an AI adjust its defaults after every single edit sounds clever. In practice it's risky. One edit might be about that project alone: the client happens to be international, or the material happens to go to an overseas review panel. If the AI turns a one-off decision into a new rule, the next local client gets bilingual handouts for no reason.

The opposite failure is just as real. If the AI never learns, people keep fixing the same thing by hand. Repetition gets tedious, tedium leads to misses, and eventually the defaults drift so far from real practice that nobody trusts them.

So the question isn't whether to learn. It's how much evidence counts. Our answer: write it down first, propose only once there's enough of it, and let a person decide.

The five steps of the loop

The loop starts with each interaction and runs all the way to a new version of the defaults. Then the new default is watched in the next round like any other. Each step has one clear job:

• Notice: whenever work is opened, moved forward or closed, the AI compares the default with the final version and records the difference, along with the reason the person gave.
• Record: the differences go into a short table, stored with a summary of the conversation in that project's own record. The original conversation is saved separately as plain text and never overwritten.
• Send back: each difference becomes a habit candidate, tagged with the project it came from, the type of work, the date and the person's own words, and is sent to a shared habit library.
• Sort: every candidate is filed under one of six kinds: goals, measures, wording, judgement, structure or process. Sorting is what makes "the same change" something you can actually compare.
• Upgrade: when the same change in the same kind reaches three occurrences, the AI proposes updating the default. Only after a person agrees does the default library move to a new version, and the old version is kept.

An example for each kind makes this easier to picture. Goals: a course project's goal keeps being changed to "run the first class" instead of "finish the materials". Measures: satisfaction keeps moving from a five-point to a ten-point scale. Wording: "trainees" keeps becoming "participants". Judgement: a certain kind of project keeps being reclassified from research to design. Structure: a "client feedback" folder keeps getting added. Process: an internal dry run keeps being added before delivery.

Why keep the original conversation separately? The difference table is the AI's summary, and summaries involve choices. If someone later asks whether that's really what was said, the original words are there to check. The summary is for everyday lookup, the original is for settling questions.

The two gates in step five work in series. The three-time threshold decides whether to ask, not whether to change. After the AI proposes, the person can still say no. The default then stays as it is, the AI records why, and next time a similar candidate appears there's a precedent to look at.

The line outside the loop: same-day write-back

The learning loop handles habits, the small edits people keep repeating. But while setting up work, people also make explicit decisions, such as adding a new type of work or adjusting a step in how new work is opened.

Those decisions don't need to wait for three occurrences. They are already decisions, so they go back into the relevant rules the same day, with a traceable entry in the change log. We draw this line outside the loop because it takes a different path: nothing is accumulated, a person simply said so.

Keeping the two paths apart keeps things legible. When you see that a default changed, you can tell whether it came from three repeats plus a sign-off, or from a specific decision on a specific day. Two sources, two kinds of record, never mixed.

Different rhythms change different layers

Learning makes a system better, but changing too often makes it hard to live with. If today's default differs from yesterday's and tomorrow's differs again, nobody knows what the current rules are. So we laid out a rhythm for what can change, and when.

• Every interaction: record the differences. No rules change.
• Weekly: list new habit candidates so people can see what the AI has noticed. Still no rule changes.
• Monthly review: held on the same day as data housekeeping. It looks at how often work was classified correctly, which defaults get edited most, which candidates have reached three, and where processes got stuck. It produces proposals, and a version only moves on a person's yes.
• Quarterly: review the types of work themselves, such as whether one should be added or two merged. This is the biggest layer, so it's looked at least often.

At the monthly review, the AI lays out its proposals as a numbered list and the person answers in one line, for example "1 yes, 2 revise, 3 later". Yes gets done, revise gets a rewritten proposal, and later comes back next month. Deciding is cheap, so people actually read the list.

Upgrades don't reach backward

A new default affects work opened from then on. Work that's already underway isn't rewritten. Whatever version of the defaults it started with, it keeps.

This looks conservative, but it's what keeps the record trustworthy. If new defaults silently rewrote old work, then looking back at a project from six months ago you couldn't tell whether its goals were set by a person or overwritten later by the system. Keeping old versions and not reaching backward lets every piece of work explain its own history.

The habit library follows the same idea. Rejected candidates aren't deleted; they're marked "not adopted" with the reason. Adopted ones link to the default version they produced. You can trace any rule back to where it came from.

The AI's job is to notice. A person's job is to say it counts.

In short

• Every time, the AI records how the final version differed from the default, and why. Original conversations are kept, never overwritten.
• A new default is only proposed after the same change appears three times. Three decides whether to ask; a person decides whether to change.
• Explicit decisions take the write-back path outside the loop, landing in the rules the same day with a record.
• Interactions record, weekly lists candidates, monthly tunes, quarterly reviews work types. Upgrades never rewrite past work, and old versions are always kept.