AI Sprint: AI Evals & Reliability — Day 2 (Testing Systems That Never Answer Twice)

Traditional software is deterministic — add(2,3) returns 5 every time, so testing is assert-equal. An LLM system returns a different, equally valid answer on every run, so 'you are not testing a value anymore. You are measuring a spread.' Evals are simply testing rebuilt for that reality: run many cases, score each against a standard, read the distribution.

The urgency is framed as four questions the room cannot answer: which model is best FOR YOUR USE CASE, by how much, is the new model actually better, and is a 1-in-1000 failure acceptable? Public leaderboards answer none of them 'because they haven't seen your data.' His refrain: 'the answer to every question tonight is testing.' The moat framing follows — everyone calls the same models; measurement is the differentiator, and 'it won't stay open always.'

Five parts, one analogy. DATASET = the test route: real customer messages ('my payment failed twice' -> billing; 'canceling, this is unusable' -> churn risk, highest severity, route to a human). GOLDEN = the correct answer — 'the right answers are golden because a human wrote them.' RUNNER = the application under test. SCORER = the examiner. REPORT = the license.

Two iron rules carry the section. Rule 1: inputs must come from REAL users, because 'real people phrase things in a way you will never phrase it.' Rule 2: goldens are human-written or at least human-checked — otherwise you are measuring 'an AI against an AI and calling it truth.' The sanctioned division of labor: AI may generate input VARIETIES; a human owns every golden. And the dataset must include negative examples — questions with no valid answer ('what's the weather on Mars?', 'what's my account balance?') — because 'a bot that makes up a balance is worse than one that says I cannot see that.'

Sizing doctrine kills the perfectionism excuse: the jump from 0 to 3 cases matters more than 3 to 50. 'A small eval beats no eval.' Start at 3-10, grow to 50 when it matters.

EXACT MATCH: free, instant, deterministic — and 'weak against unstructured text' (the 5x5 answer above fails a string compare while being right). The fix is upstream: force the model to answer in fixed-schema JSON, which converts open text into exact-matchable fields — 'the cheapest quality lever there is, and almost nobody pulls it.' The scorer taxonomy runs Boolean, scale, label, deterministic code, LLM-judge, with one rule: work DOWN the list — '2+2 is 4, don't ask an LLM.'

LLM-AS-JUDGE: for what code can't check — tone, politeness, helpfulness. Demoed on an angry customer message with a criteria rubric. Four design rules: force structured judge output (1/0, not a paragraph); give an escape hatch ('if you can't tell, say so'); write a real rubric, not 'is this good?'; and the judge should be cheaper/lighter than the model it judges.

TRAJECTORY: for agents, grade the DECISION PATH, not the answer — did it call lookup_order before issue_refund, did it check eligibility, 'did you check the mirror before you signal.' The threat it catches is the false positive: an agent that reaches the right answer without verifying — 'right answer for the wrong reason' — which exact match happily passes. The advanced form keeps a hand-maintained substitute-tool list (tools that legitimately do the same job) and optionally a judge assessing tool-intent equivalence, closing with a cost/hallucination-risk/best-use comparison table of all three.

The reframe: 'the mistake is thinking a judge certifies correctness. It does not. It never claimed to.' If you need correctness, use a deterministic check. A judge is a MEASUREMENT INSTRUMENT, and instruments get calibrated: the honest question is 'how often does the judge agree with a human?'

The calibration protocol, four steps: (1) build human ground truth — 5 raters across 40 outputs; (2) run the candidate judge on the same set; (3) report the agreement rate — his worked example: 174/200 = 87%, producing the boardroom-defensible sentence '87% agreement across 200 human-labeled cases'; (4) recalibrate every time the underlying model changes. Two structural upgrades: a JURY of judges 'almost always beats a single judge... independent errors partly cancel out,' and never let the judge share a model family with the generator — self-recognition bias makes models favor their own outputs.

He grounds the pattern's legitimacy in production reality: this exact architecture runs today at DoorDash, Uber, and Dropbox, publicly documented (his claim, flagged as dating).

The practice he names as the single most important takeaway. A failed driving test doesn't return a percentage — it returns a breakdown sheet. The loop: run the eval -> READ every failure ('actually read them') -> cluster the failures -> NAME each cluster -> count them -> fix the LARGEST cluster -> rerun. Priority is by count, not by which failure annoys you.

Live on the support bot: 12 cases, 8 passed, 66.7%, three named clusters — including a severity underrate ('the app crashed and I lost an hour of work' scored 4, humans say 5) and an out-of-scope miss ('what is my account balance' answered instead of refused). Then the compounding rule that turns debugging into an asset: every fixed bug 'become a permanent test route' — a new dataset row forever, so the regression 'can never silently come back.' The failure-mode catalog doubles as the handoff artifact a departing developer owes the next one.

Three extensions, briefly but sharply drawn. RAG: evaluate retrieval ('did it open the right page?') SEPARATELY from generation ('did it answer from the book or make it up?') because the two failures 'need completely different fixes' — a retrieval fix is chunking/embedding work, a generation fix is prompting/grounding work.

MULTI-TURN: a bot can pass every individual turn and still fail the conversation — so conversational systems need SESSION-LEVEL grading on task completion, resolution, and recovery, not per-turn scores alone.

ADVERSARIAL: a deliberate attack suite — prompt injection, jailbreaks, PII leakage, tool misuse, off-topic drift — with the stakes escalating by system type: 'a chatbot that gets injected says something embarrassing. An agent that gets injected takes an action.'

EVAL OVERFITTING is the driving-school scandal: 'you taught it the route. You have not taught the learning to the system.' Tuning against a fixed set optimizes for those 50 cases — 'you did not build a better system, you built one that's good at your 50 cases.' Four mitigations: hold out a test set you never tune against; refresh the dataset from live traffic; delete production leakage out of the eval; grow the dataset from newly discovered failures (which the error-analysis loop does automatically).

EVALS VS GUARDRAILS is the ship-line distinction: 'an eval is a driving test. Guardrail is the examiner's second brake pedal.' Eval: before ship, fixed dataset, minutes, outputs a score and a ship/no-ship decision. Guardrail: after ship, live traffic, MILLISECONDS, and it acts — blocks, rewrites, escalates to a human. The latency budget dictates the tooling: 'you cannot use a slow LLM judge as a guardrail' — guardrails run on exact match, schema validation, tool-call allow-lists, PII redaction, refusal detection, and cost ceilings.