TLDR; The thing that keeps a skill or agent effective when the model changes is not a better prompt, it is a regression eval that runs on every change and a human gate at the point where being wrong is expensive. Pin the model, grade each change against a fixed set of real cases with cheap deterministic checks, and let a human approve anything that becomes a fact of record. A model swap is a migration, not a config flag, and minor updates break production more reliably than major ones.
A compliance assistant I worked on has one job it must never get wrong: quote the right price. The training provider behind it publishes a government-funded course fee of $4,500 for eligible students, and a full fee of $9,950 for everyone else. Get that backwards on a live enrolment page and you have misquoted a regulated price to a prospective student. The model does not know which number applies. It will happily produce either, and both look equally confident.
So the assistant does not ask the model for the price. It reads the price from a fact store that a human has signed off, and the test that guards it is blunt: given the eligible-student question, the answer must contain $4,500 and must never contain $9,950. That is not an LLM grading another LLM. It is a substring check, cheap and reproducible, that fails the build the instant the assistant quotes an un-approved figure.
That guardrail is the whole discipline in miniature. When a model is updated or swapped, the prompt that used to behave stops behaving, and the only way you find out before your users do is a test that runs on every change and a human standing at the gate for anything that matters. This guide is how I do that on two projects: Knowledge Navigator, a local-first memory system I build for myself, and the compliance assistant above, which I keep generic here because it is client work.
Deterministic ground truth first, the judge last
The instinct when you need to grade AI output is to reach for another AI to grade it. Resist it for as long as you can.
The reason is measurable, but task-specific. Eugene Yan’s survey of LLM evaluators reports a correlation of 0.8 to 0.9 between individual human experts and the expert average, against 0.3 to 0.6 between GPT-3.5 and humans, on particular summarisation evaluations; it also cites factual-consistency tasks where older model evaluators caught only 30 to 60 percent of defects. Those ranges are not universal limits for current judges. They are evidence that evaluator performance must be measured on the actual task rather than assumed. The survey also catalogues position bias, verbosity bias, and self-preference. A judge can be useful, but it is not ground truth.
So on both projects the grader hierarchy runs cheapest and most reliable first: exact match, then field diffs, then rule-based and substring assertions, and only then an LLM judge for the things nothing else can grade. The compliance assistant never graduates past substring assertions on purpose, because keeping the gate cheap and reproducible matters more than grading nuance on a price lookup. Knowledge Navigator is a retrieval system, so its primary metric is Recall@k: for a known question, did the right memory come back in the top k results. That is a set-membership check against a labelled answer, not a judgement call. As Hamel Husain puts it in his LLM-as-a-judge guide, creating an LLM judge is “a nice ‘hack’ I use to trick people into carefully looking at their data!” The discipline is the looking. The judge is downstream of it.
Every miss becomes a permanent test
The loop most agent teams are missing is the one conventional software has had for decades: a bug becomes a failing test, the test fails until the bug is fixed, and the test stays in the suite forever so the bug can never come back. Applied to LLM failures, that is the eval flywheel. A production miss gets triaged, reduced to a minimal repro, given a grader, and added to the dataset that gates every future change.
Knowledge Navigator’s suite grew this way, one caught failure at a time, from 45 tests to 74. Each new test is a failure I would otherwise have shipped twice. One of them, case E-7, exists because a symlink-flattening bug once collapsed distinct memories into one during indexing; the test now asserts the structure survives, and no future model or refactor can quietly reintroduce it. The compliance assistant has an xfail for every fact that does not yet exist in the store, which pins the correct behaviour: when asked about a course that is not on file, the assistant must decline, not invent one.
Two lessons from the research hold up in practice here. The first is that error analysis comes before infrastructure: you read real traces and let the failure categories emerge before you build any grading machinery, because you cannot write good evaluators for a failure surface you have not seen yet. This is the opposite of test-driven development, and it is why “we will add evals later” is the most expensive sentence in the field. The second is counterintuitive: a 100 percent pass rate is a warning sign, not a victory. It usually means the suite is too easy and has stopped challenging the system. A suite that stresses the agent honestly sits lower, and binary pass or fail beats a 1-to-5 quality score, because the gap between a 3 and a 4 is an argument nobody wins twice the same way.
A model swap is a migration, not a config flag
Here is the failure that motivates the whole guide. You do not decide to break your agent. A provider ships a minor update, or you move from one model to a cheaper one, and behaviour you spent weeks tuning shifts underneath you.
Tian Pan’s Semver Lie names the trap precisely: a version number on a model is marketing, not a compatibility contract. The output distribution is the real API surface, and it moves with every weight change. Counterintuitively, minor updates break production more reliably than major refactors: they get less scrutiny and less documentation, and aliases push them out automatically. The damage shows up as “negative flips,” Apple’s term for cases the old model got right and the new one gets wrong, which hide comfortably under an aggregate benchmark that went up. Never auto-adopt the latest anything.
Both my projects pin the model for this reason. The compliance assistant pins a specific cheap model as its default and exposes a --model override so I can run the same fixed suite against a candidate before committing to it. That is the same staged-rollout thinking you would apply to a database migration: pinned eval, then the new model on the side, then a careful promotion, then a rollback path that actually works.
Knowledge Navigator adds a wrinkle most migration plans forget: it has a second model in the loop, the embedding model that turns memories into vectors. Swapping it, in my case moving from a small general embedder to a stronger multilingual one, is not a prompt change, it is a full re-embed of every stored memory, because old and new vectors are not directly comparable. I do not assume a standard overlap rate between embedding models; I measure top-K overlap and Recall@k on the labelled corpus. Any divergence needs review, and if recall drops more than the project’s allowed 2 percent, the migration does not ship. The regression gate is the thing that makes the swap safe to attempt at all.
AI proposes, a human disposes
Automation is allowed to do a great deal in these systems. It is not allowed to have the last word on anything expensive to get wrong.
The compliance assistant draws the line at the fact store. Automation may propose a change, it can open a pull request adding or correcting a fact, but it may never commit a compliance fact on its own. A human approves the record, because the record is what the assistant will quote as true tomorrow. Knowledge Navigator draws the same line at indexing: the system proposes structure, and a human verifies it, encoded as a literal [Human] task in the spec rather than left to good intentions. Structure proposes, a person disposes.
That is one of three oversight modes worth choosing deliberately rather than by accident. Human-in-the-loop means a person decides each action, appropriate when a wrong action is costly and rare. Human-on-the-loop means the agent acts and a person can veto, appropriate for higher-volume, lower-stakes work. Human-out-of-the-loop means autonomous within pre-set bounds, appropriate only for paths you have evalled hard enough to trust. Training-time alignment and this runtime oversight are complementary, not substitutes: alignment lowers the rate of bad decisions, it does not eliminate the individual one that misquotes a price.
One thing not to use as your escalation trigger: the model’s own confidence. Raw confidence scores are unreliable as a signal for what to hand to a human, which is why the compliance assistant escalates based on the category of the action, quoting a price, committing a fact, rather than on how sure the model claims to be. A standing share of cases will always route to a human queue, and that is a healthy equilibrium, not a failure to reach 100 percent.
What else the harness pays for
Once you have a regression suite and real traces in front of you, it earns its keep well beyond catching drift.
It calibrates thresholds you would otherwise guess. Knowledge Navigator has a relevance score floor that decides what counts as a match; I had it set to 0.80 and the suite showed that floor was inert, letting weak results through, and that 0.86 was the value that actually separated good retrievals from noise. I would not have found that number by intuition. It surfaces the biases in your own judge, since the same labelled set that grades output can be used to check whether the judge is drifting toward the longer or the first answer. And it turns a scary model upgrade into a capability audit: point the fixed suite at the new model and you learn what it does better, not just whether it regressed.
It is worth saying what does not need any of this. I also maintain a charting library, deterministic code with a deterministic test suite and no model anywhere in the loop. It has zero evals of the kind described here, correctly, because nothing about it can drift. That is the real precondition, and it belongs in the when-to-use-AI decision: evals and human oversight are the tax you pay for putting a probabilistic component in a place where being wrong has a cost. If there is no model in the loop, or no cost to being wrong, do not build the harness. If there is both, build it before you ship, not after.
The honest state of it
I want to be accurate about where this actually stands, because the failure mode of a guide like this is to describe a finished machine. Knowledge Navigator’s golden baseline is still being built: the labelled answer set is not yet frozen, and there is no committed baseline file to diff every run against, which means the regression gate is real in intent but not yet fully automatic. That is normal. Evals are not a feature you finish, they are a discipline you keep compounding, one caught failure at a time, and the suite that guards a mature agent is mostly a museum of mistakes it is no longer allowed to make.
The static gate is never finished for a deeper reason, too. Shankar and colleagues named it “criteria drift” in Who Validates the Validators: you cannot fully define your grading criteria before you see outputs, because seeing outputs is what teaches you the criteria. A CI eval gate is a snapshot of last month’s understanding, and it needs a human refreshing it as the product and the failures evolve. That is the point that ties the whole thing together. The eval catches the drift in the model. The human catches the drift in the eval.
I build a harness like this into every Applied AI engagement. An agent without one is a demo.
Related reading
- When to use AI, and when not to, the decision that comes before you build any of this
- Quality gates, session reviews, fact-checking, and code review, the writer/reviewer sibling of the eval harness
- The memory hierarchy, how retrieval systems like Knowledge Navigator are structured
- Multi-agent orchestration, where independent review agents fit the same separation-of-concerns idea
Sources
- AI Evals: Everything You Need to Know, Hamel Husain and Shreya Shankar, September 2026. The evals FAQ: grader hierarchy, error analysis first, binary over Likert. Practitioner reference
- Using LLM-as-a-Judge For Evaluation: A Complete Guide, Hamel Husain, October 2024. Critique shadowing, pass/fail judgements, and the judge as a device for making people look at their data. Practitioner reference
- Evaluating the Effectiveness of LLM-Evaluators, Eugene Yan. Survey of LLM-as-judge alignment, low defect recall, and named biases (position, verbosity, self-preference). Practitioner synthesis
- Who Validates the Validators?, Shreya Shankar et al., UIST 2024. The criteria-drift finding: grading outputs is what reveals the criteria, so a static gate is always a snapshot. Peer-reviewed research
- The Semver Lie, Tian Pan, 2026. Why a minor model update breaks production more reliably than a major refactor, and negative flips under rising benchmarks. Practitioner analysis
- The Model Migration Playbook, Tian Pan, 2026. Treating a model swap as a staged migration: pinned eval, shadow traffic, canary, rollback. Practitioner analysis
Common questions
How do you stop an AI agent from breaking when the model is updated?
Treat every model change as a migration, not a version bump. Pin the model, keep a regression eval that runs on every change, and gate the swap on that eval. If retrieval or answer quality drops past a set threshold, the swap does not ship. A minor update needs the same check as a major one, because minor updates break production more often than people expect.
Are LLM-as-judge evals reliable enough to catch errors?
Only within limits. An LLM judge agrees with human raters far less than two humans agree with each other, and it misses a large share of real defects. I use deterministic graders first, exact match, field diffs, substring assertions, and reserve the judge for things the cheap checks cannot grade. The judge is a development tool, not production truth.
Where does the human belong in an AI agent workflow?
At the point where being wrong is expensive. Automation can propose a change, draft an answer, or open a pull request, but a human approves anything that becomes a fact of record. For lower-risk work the human moves to on-the-loop, able to veto rather than approve each step. Raw model confidence is a poor trigger for what to escalate, so the escalation rule is the risk of the action, not the model’s self-reported certainty.