Somewhere in your organization there is a list of places AI already works: marketing drafts, meeting summaries, first replies to routine tickets, the supporting paperwork around sales and HR. It grew on its own, without a steering committee. Now try to write the second list: the mission-critical processes where a model runs. The credit decision. The regulatory filing. The payment run.
I tried to write that second list for my own field and it came out short. That bothered me, because the obvious explanation does not hold. The models handling the marketing drafts are the same models that stall at the credit decision. Yes, a serious deployment would wrap the model in grounding, constraints, and checks. That wrapping is exactly the point: it is process work, and nobody has done it. Whatever separates the two lists, it is not capability.
Look closer at the first list and the real difference shows itself: every item on it is work where a wrong answer costs an edit. A bad draft gets rewritten. A weak summary gets ignored. Nobody had to design anything before switching the tool on, because every mistake could be taken back. The second list is different in exactly one way. Each of those processes has a step that cannot be taken back.
The Step You Can’t Take Back Makes Two Demands
A process with an irreversible step demands two things from whatever performs it:
- Do the same thing the same way twice. Repeatability. A draft can vary between runs; a credit decision cannot.
- Say why, a year later. Explainability. To an auditor, a regulator, a client on the phone.
A bare model gives you neither. Repeatability can be engineered, but it is a property of the system you build, not of the model you buy. Ask a model why it decided something and you get a fluent story, written after the fact. And that assumes you can still ask: rented models get replaced, rerouted, and silently updated, so a year later the exact model that made the decision may no longer exist to be questioned. In a draft none of this matters. At a payment run it is disqualifying.
“Irreversible”, defined practically: payments get clawed back and filings get amended – slowly, expensively, in public. The line runs where taking it back stops being an edit and becomes an incident. These two demands are older than AI. Reproduce the decision, defend it later. Every serious control function was built to answer exactly this.
So the work of getting AI into serious processes was never model selection. Someone has to build, around the model, what the model does not carry inside. The people doing this work have names for the layer by now: the harness, the skills, the workflows, the loops, the goals. Whatever you call it, it has to answer three things:
- What gets verified. The specific claims a wrong outcome would trace back to, checked every run. The check is your repeatability.
- Against which record. A source the model didn’t write. The record will not tell you what the model was thinking; it proves what was checked before the step ran, which is the answer an examiner actually accepts. If no such record exists, that build comes first.
- Who owns the go. A named person at the irreversible step. Not a team, and not the machine.
The schema above is my own lab, drawn from life: a model drafts a news briefing there daily, and the drafting was solved in a week. The real effort went into the other two boxes:
- every factual claim is checked against the stored original source, never against the model’s memory of it;
- a failed check blocks the release instead of flagging it politely;
- the checker’s rules are mine, in writing, so “why did this ship” has an answer I can show.
The model was the cheapest part, and the only part I could swap tomorrow without redesigning anything.
The Pile That Answers Questions
The checkpoint assumes one thing: a record worth checking against. That assumption fails quietly, and often. A department sits on years of accumulation: mail threads, slide decks, exports from dead systems. Nobody can describe how the process runs. The pile is the process. Then a model gets pointed at the pile, and the pile starts answering questions. Three confident paragraphs on demand. It looks like the structure problem solved itself.
What got deployed is a summarizer. Every summary drops details it cannot know matter: the exception, the changed date, the condition in the second attachment. The summaries get saved and forwarded and become sources themselves.
The pile grows smoother and larger and quietly less true. This holds up as long as questions stay small. Ask what was promised to whom across two years and forty threads, and there is nothing underneath. Those connections were never recorded, and no amount of reading window makes a model see what was never written down.
The pile fails for a reason worth naming, because it is the same reason the record layer has to exist at all. Today’s models do not remember, and they do not understand. They read. Every session starts from zero: whatever the model appeared to know about your organization yesterday left with yesterday’s session. What gets sold as “memory” is retrieval: notes fetched back into the model’s view at the moment you ask. A filing cabinet with a very fast librarian. And knowing something does not mean understanding it. Give a new hire every file in your archive and perfect recall: on day one they know everything and understand nothing. Understanding arrives by working the cases, remembering what broke, carrying the why from one decision into the next. The model is permanently on day one.
That is the real ceiling on where models can be applied today. Any process whose value depends on accumulated understanding – the client’s full history, the precedent from March, why the exception exists at all – cannot be handed to a model alone, because the model holds none of it. The remembering has to live in the process: in records kept at the moment things happen, in the relations written down between them. Build that, and the model becomes genuinely useful on top of it. Skip it, and you get the pile.
One Afternoon, Two Checks
Pick the process you most want AI inside. Ask two people who run it to describe it, separately, in writing. Then mark its first irreversible step and try to name who owns it.
The rule for this was already on my own shelf. Every organization has always had workers whose exact steps could not be written down in advance: the crisis manager, the negotiator, the senior expert handling the case nobody has seen before. We never governed them with instructions. We governed them with a defined goal, hard limits, and outcomes checked against a record. That is the category AI belongs to. It is good at exactly the work you could never write full instructions for, and it needs what your best expert always needed: a clear goal, real limits, and someone who checks the outcome.
Which means the question was never whether a process is too critical for AI. The question is whether your outcome-checking has grown up enough to hold a worker you cannot fully script. Growing it is slower than buying a license, and it starts with two people writing down what they think the process is.
One last distinction, because it decides who you put on this. Implementing AI is engineering, and engineering has become cheap: the models increasingly write their own plumbing. Designing what holds the AI is architecture: where the checkpoint sits, what the record captures, who owns the go, what escalates when. That work got no cheaper, and it needs a rarer profile: someone who can design the control and understands how the system will actually enforce it. A designer who cannot see the mechanism writes rules that do not bind. A builder without the governance ships something that runs and answers to nobody.
The profile is rare for an organizational reason, not a personal one. In most companies these two halves live in different tribes: governance speaks the legal language, the mechanism speaks data and IT, and the two meet mostly in review meetings where each finds the other slightly unreasonable. AI adoption is asking those tribes to produce one person’s worth of judgment together. The companies that adopt fastest will not be the ones with the best engineers. They will be the ones where somebody translates. Deciding what to grow first, and in what order — that is architecture, and it is the work.