Evals and drift: keeping agents honest when the model changes

TLDR; Two things keep an AI agent working when the model underneath it changes: a regression eval (a fixed set of tests, rerun on every change, that catches cases that used to work and now don’t) and a person who approves anything where being wrong is expensive. Pin the model to an exact version, and grade each change against real cases with cheap checks that give the same answer every time. Treat a model swap as a planned migration: minor model updates break live systems more reliably than major code rewrites, because they get less scrutiny.

Why agents break when the model changes

A compliance assistant I worked on has one job it must never get wrong: quote only prices a person has approved. Say a training provider’s approved fee for a diploma is $5,000, and a proposed increase to $6,500 is still waiting for sign-off (both figures are illustrative). The model can’t tell which figure is approved, and it will quote either with equal confidence. Quote the wrong one on a live enrolment page and you have misquoted a regulated price to a prospective student.

So the assistant never asks the model for the price. It reads it from a fact store, a list of facts each marked approved or still proposed, and it may quote only the approved ones. The test that guards it is blunt: asked the fee question, the answer must contain $5,000 and must never contain $6,500. That text check is cheap, gives the same result every time, and stops the release the moment the assistant quotes an unapproved figure.

When a model is updated or swapped, a prompt that used to work can start giving worse answers. That is model drift, and the reliable way to find out before your users do is a test that runs on every change, with a person approving anything that matters. The four principles below come from two projects. One is Knowledge Navigator, a searchable memory store I build for myself. The other is the compliance assistant, client work, so I keep it anonymous here.

Grade with fixed-answer checks first, and an AI judge last

Why: an LLM judge (LLM-as-judge: one large language model, the kind of AI behind ChatGPT and Claude, grading another’s output) is less reliable than it looks, and you only learn how much less by testing it on your own task. Eugene Yan’s survey of LLM evaluators reports that a single human expert agreed with the panel’s average at 0.8 to 0.9, on a scale where 1.0 is perfect agreement. GPT-3.5, an older OpenAI model, managed 0.3 to 0.6 against humans on particular summarisation evaluations. On checks of whether a summary matches its source, older LLM judges caught only 30 to 60 percent of defects. Those ranges come from particular tasks.

The survey also names three biases: a judge can favour an answer for its position, for its length (verbosity), or because it wrote it (self-enhancement).

In practice: both projects run the cheapest, most reliable check first: an exact match, then a field-by-field comparison, then rule and text checks, and an LLM judge only for what nothing else can grade. The compliance assistant never goes past text checks, on purpose. Knowledge Navigator is a search system, so its main measure is Recall@k: for a question with a known answer, did the right memory come back in the top few results (the top k)? That is a yes-or-no check, with no judgement in it.

Do this: write the cheapest check that could catch the failure. Reach for a judge only when nothing cheaper can grade the output, and test the judge against answers a person has marked right or wrong before you trust it. As Hamel Husain puts it, building a judge is “a nice ‘hack’ I use to trick people into carefully looking at their data!” The looking is the point.

Turn every miss into a permanent test

Why: conventional software has handled regressions the same way for decades. A bug becomes a failing test, and the test stays in the test suite, the full set of tests, so the same bug can’t return unnoticed. Applied to an AI model, the same loop is often called the eval flywheel. Take a failure from real use, cut it down to the smallest example that still fails, write a check for it, and add it to the test set every future change must pass.

In practice: Knowledge Navigator’s test suite grew from 45 tests to 74, one caught failure at a time. Case E-7 exists because a bug with symbolic links (file shortcuts) once merged separate memories into one when the search index was built; the test now checks they stay separate. The compliance assistant marks some questions as expected failures (xfail): the answer it should give depends on a fact still waiting for sign-off. The run records them without failing, and they start to pass once a person approves the fact. An ordinary test covers a fact the store doesn’t hold at all: asked about funding it has no fact for, the assistant must refer the student to the admissions team and invent nothing.

Do this:

• When a case fails in real use, make it a test before you fix it.
• Read real traces, records of the agent at work, before you build any grading. Let the kinds of failure show themselves first; it is hard to write a good check for a failure you haven’t seen. In my experience, “we will add evals later” is the sentence that costs teams most.
• Grade pass or fail. Pass-or-fail labels come out more consistent than a 1-to-5 scale, where the line between a 3 and a 4 is hard to pin down.
• Distrust a 100 percent pass rate. It probably means the suite isn’t challenging the agent enough.

Treat a model swap as a migration

Why: Tian Pan’s Semver Lie argues that a model’s version number is marketing, not a promise that nothing will break. What the model says is what your system depends on, and it moves every time the model is retrained. In his account, minor updates break live systems more reliably than major code rewrites, because they get less scrutiny. Model aliases (names like opus that always point to the latest version) push them out automatically. The damage shows up as “negative flips”: cases the old model got right and the new one gets wrong, hidden under an overall score that went up. The term comes from model-update research, and Apple’s MUSCLE paper applied it to LLM updates.

In practice: both projects pin the model, fixing the exact version they call. The compliance assistant pins a cheap model, and a --model setting lets me run the same suite against a new model before I switch. Knowledge Navigator has a second model to pin, the embedding model that turns each memory into a list of numbers for search. Replacing its small general model with a stronger multilingual one means re-processing every stored memory, because the old and new numbers can’t be compared. On the questions with known answers I measure Recall@k and how many top results the two models share; if recall drops more than the project’s allowed 2 percent, the migration does not ship.

Do this: never auto-adopt the latest model; the model selection guide covers choosing one. Stage it like any risky system change: run the eval on the pinned model, run the new model alongside it on copies of real requests, send a small share of real requests to it first, and keep a tested way to switch back.

AI proposes, a human disposes

Why: automation should not have the last word on anything expensive to get wrong. Training a model to behave well lowers the rate of bad decisions; it doesn’t stop the one that misquotes a price.

In practice: the compliance assistant draws the line at the fact store. Automation may propose a new or corrected fact for review but never saves one; a person approves the record, because the record is what the assistant will quote tomorrow. Knowledge Navigator draws it at indexing: the system proposes how memories are grouped, and a person checks it. That check is a [Human] task in the project plan, so it never depends on someone remembering. The quality gates guide applies the same split to code and prose review.

Do this: choose the oversight mode for each action on purpose.

Send a case to a person based on the kind of action. A model’s self-reported confidence tends to run high, so the compliance assistant hands a case to a person whenever it is about to quote a price or save a fact. Some cases will always land with a person, and they should.

What else the harness gives you

Once you have the harness, the regression suite plus the human gate, it does more than catch drift:

• Thresholds set by evidence. Knowledge Navigator’s minimum relevance score, the cut-off below which a search result is dropped, was 0.80; the suite showed it let weak results through, and that 0.86 separated good results from noise. I would not have found that number by intuition.
• A check on your judge. The questions with known answers also show whether the judge is drifting toward the longer or the first answer.
• A capability audit on every upgrade. Point the fixed suite at a new model and you learn what it does better as well as whether it regressed.

When you don’t need it

A model can answer differently each time. Evals and human oversight are the price of using one where being wrong is expensive. I also maintain a charting library: its code gives the same output for the same input, ordinary tests cover it, and no model is involved. It has no evals of this kind, and needs none, because nothing in it can drift. If there is no model in the loop, or no cost to being wrong, don’t build the harness; if there are both, build it before you ship. The when-to-use-AI guide covers that decision.

Limits

• A gate is never finished. Your grading criteria keep changing; Shankar and colleagues call this “criteria drift” in Who Validates the Validators. You learn the criteria by seeing outputs, so an automated eval gate is a snapshot of last month’s understanding. A person has to keep the eval itself up to date.
• Mine isn’t finished either. Knowledge Navigator’s set of questions with known answers is not yet final, and there is no saved baseline to compare each run against. Its regression gate is designed but not yet fully automatic.

I build a harness like this into every Applied AI engagement. An agent without one is a demo.

Related reading

• When to use AI, and when not to, the decision that comes before you build any of this
• Quality gates, session reviews, fact-checking, and code review, the writer/reviewer sibling of the eval harness
• Claude Code memory, how systems like Knowledge Navigator store information and find it again
• Multi-agent orchestration, where independent review agents fit the same separation-of-concerns idea

Sources

• Evaluating the Effectiveness of LLM-Evaluators, Eugene Yan. Survey of LLM-as-judge alignment, low defect recall, and named biases (position, verbosity, self-enhancement). Practitioner synthesis
• Using LLM-as-a-Judge For Evaluation: A Complete Guide, Hamel Husain, October 2024. Critique shadowing, pass/fail judgements, and the judge as a device for making people look at their data. Practitioner reference
• AI Evals: Everything You Need to Know, Hamel Husain and Shreya Shankar, September 2026. The evals FAQ: grader hierarchy, error analysis first, binary over Likert. Practitioner reference
• The Semver Lie, Tian Pan, 2026. Why a minor model update breaks production more reliably than a major refactor, and negative flips under rising benchmarks. Practitioner analysis
• MUSCLE: A Model Update Strategy for Compatible LLM Evolution, Echterhoff et al., Apple, EMNLP Findings 2024. Negative flips in LLM updates; the term itself comes from Yan et al., Positive-Congruent Training, CVPR 2021. Peer-reviewed research
• Can LLMs Express Their Uncertainty?, Xiong et al., ICLR 2024. LLMs verbalising their confidence tend to be overconfident. Peer-reviewed research
• Who Validates the Validators?, Shreya Shankar et al., UIST 2024. The criteria-drift finding: grading outputs is what reveals the criteria, so a static gate is always a snapshot. Peer-reviewed research
• The Model Migration Playbook, Tian Pan, 2026. Treating a model swap as a staged migration: eval suite, shadow traffic, canary, rollback. Practitioner analysis