The models were not the only thing under evaluation. Our harness was wrong too.
LLM Personalities, part 3 of 3: lessons from building the evaluation, July 2026.
We started with a familiar question: which model should do the job? By the end, we were spending almost as much time evaluating the evaluation.
The time was well spent. An agent score is the output of a system: model, prompt, serving stack, tools, knowledge snapshot, judge, gates, state restoration, and reporting code. If any one moves between lanes, the leaderboard gains false precision.
A perfect answer can fail the wrong gate
Our semantic judge could mark an answer fully grounded while a deterministic gate rejected its citation. The agent cited exactly what search returned, but search sometimes returned a derived fact while the fixture expected its source document.
The model had done the right thing. Two parts of the evaluator disagreed about identity.
This appeared in roughly two to five tasks per run and added an estimated 13–30% noise to strict pass rates. We corrected the concept before the code: score content quality separately from delivery-contract compliance, then resolve citation lineage before declaring a source wrong.
An efficiency budget is not a quality verdict
Claude produced grounded answers and still lost points for reading too much evidence. A context-token budget was allowed to veto the result, punishing the best-grounded models with a cost metric.
We split the scoreboard. Quality asks "did it work?" Efficiency asks "what did it cost?" After we tightened the production contract, Claude went from 1.63 to 1.2 tool calls per task without losing grounding. That is useful optimisation, and a combined pass gate would hide it.
"Nothing is moving" does not mean "finished"
An early ingest drain check watched counters until they stopped changing, then called the lane settled. A worker that has not had a chance to start also produces a flat line, so a stalled system can look completed.
The reliable gate needs positive evidence: expected work observed, queues drained, workers healthy, then a bounded quiet period. "Stopped moving" means "done" only when the mover was guaranteed a chance to move.
Identical data did not mean identical searchable state
Each lane restored the same frozen snapshot. During replay of about 17,000 keys, the live watcher could hit a roughly one-second period with no direct-read responders. Around 22–24 entity-summary events could be missed, leaving a small random gap in the in-memory projection.
Identical source data still produced different searchable state.
The fix is retry with backoff and a readiness check that verifies the derived projection, not merely the source store. A benchmark reset must validate every view its workload reads.
Small suites make big stories from small differences
Ten tasks repeated three times exposed systematic behaviours: skipped tools, unsupported claims, broken prompts. They could not confidently order candidates separated by a few points; confidence intervals were about ±17 points.
We now repeat tasks, analyse paired results, publish uncertainty, and call a tie a tie. A task nearly every model fails is treated as a retrieval or fixture defect until proven otherwise. One early fixture demanded one source identifier even though valid near-duplicate sources existed; the system itself had made compliance ambiguous.
Operations can rewrite a leaderboard
- A reranker took about 6.4 seconds cold against a 6-second timeout, intermittently changing retrieval. Evaluation raised the bound to 15 seconds.
- Interrupted append-only runs left duplicate JSONL records. Lane files now start clean and reports keep the last task/repeat record.
- Starting a development stack could recreate a dependency and replace supposedly frozen state.
- The judge stays pinned to one remote model, never the candidate and never a local model competing for the same GPU slot.
None of these bugs lives in "the model," and every one can change its score.
The checklist we use now
- Freeze and fingerprint the source snapshot.
- Pin the tool contract, prompt, inference settings, reranker, and judge.
- Probe the serving stack before each lane.
- Verify derived state after restoration.
- Require positive evidence before asynchronous work is considered drained.
- Repeat tasks, report confidence intervals, and retain raw attempt records.
- Separate quality, contract compliance, latency, and cost.
- Investigate universal failures as harness defects.
- Re-run finalists against the exact production contract.
The goal is a benchmark whose mistakes are observable and whose claims stay smaller than its evidence.
We did choose models. More usefully, we learned which surrounding systems had been impersonating model behaviour. Before asking why an LLM failed, make the harness prove the LLM was what it measured.
Method notes: agent bake-off: 10 tasks × 3 repeats per lane over a frozen snapshot, identical tool loop, judge pinned to deepseek-v4-flash. Ingest evaluation: 15 tasks × 3 repeats. Quality and efficiency were retained separately; production-contract figures were rerun after known gate defects were corrected.
LLM Personalities: Part 1: The briefing mattered more than the hire · Part 2: One knowledge base, two AI jobs · Part 3: The evaluation harness was wrong too
Related reading: The story of making local agents usable · Hardware, engines, and benchmark results