One knowledge base, two AI jobs, and the smaller local model won one
LLM Personalities, part 2 of 3: an ingest-model bake-off, July 2026.
Levirge Brain made a single-model decision look reasonable: an LLM helps build the knowledge base, then an LLM searches it and answers questions. We assumed the strongest agent belonged in both seats. The ingest bake-off proved otherwise.
The agent seat is interactive: understand a question, search, inspect evidence, and produce a grounded answer. The ingest seat is editorial: turn raw documents into concise facts, useful entities, and wording another model can retrieve later.
We had already tested the first seat. For the second, we rebuilt the same knowledge snapshot with different ingest models, then asked a fixed agent the same 15 questions three times. The answering model and judge stayed pinned. Only the model preparing the knowledge changed.
The smaller model produced the most useful knowledge
| Ingest lane | Strict pass | Content pass | Tier A time |
|---|---|---|---|
| Ornith 9B local | 26/45 (58%) | 36/45 (80%) | 1,398s |
| Gemma 26B hosted | 26/45 (58%) | 34/45 (76%) | 1,786s |
| DeepSeek v4-flash baseline | 22/45 (49%) | 33/45 (73%) | about 30 min/pass |
The differences sit within the 95% confidence interval, so "won" needs care, and the result says nothing about Ornith being smarter overall. We selected it because it tied on aggregate, ran locally, and uniquely made one stubborn test, development guidance about testability in one subsystem, pass when all nine DeepSeek and Gemma attempts failed. It also produced perfect content on another task in every repeat.
The graph did not get bigger. Candidate stores had the same shape: about 1,055 entities and 5.7 extracted facts per document. The wording and retrievability of the knowledge inside it changed.
Local versus hosted was not the story
Gemma gave us a useful twin control. Local Q4 and hosted higher-precision Gemma produced the same content outcome in the comparable runs. The local Tier A lane was also fastest at 862 seconds.
Quantisation can still matter elsewhere. For this workload, editorial choices mattered more than where the weights were served, and blaming local inference would have sent us in the wrong direction.
"Strict pass" measured an implementation detail
Search could return a derived fact's identifier while the fixture expected its parent document's identifier. The answer could contain exactly the required evidence and earn a perfect semantic judgment, yet fail the deterministic citation check.
That affected roughly two to five tasks per run, enough to add 13–30% noise to a 15-task scorecard. We retained strict pass because the delivery contract matters, and added content pass because model quality should not be confused with an identifier-lineage defect.
A full rebuild is a migration
The evaluation subset was 16 documents and took about half an hour per lane. A full re-ingest was 383 documents, followed by 1,197 entity summaries. Documents completed in minutes; consolidating the graph took about 12 hours, roughly 24 seconds per entity.
Updating the ingest model is a data migration, not a configuration toggle. It changes stored facts and summaries, consumes hours, and needs frozen inputs, verification, and rollback thinking.
The subset still agreed with the full run on 13 of 15 tasks. It was cheap enough to reject weak candidates before a corpus rebuild, but no substitute for validating the eventual store.
We ended with two seats
- Answering agents are selected for tool use, grounding, latency, cost, and tolerable failure mode.
- Ornith 9B runs locally for categorisation because it produced the strongest content result here and keeps recurring ingest on-premises.
- A hosted model remains a fallback; local-first is an operating choice, not a purity test.
The broader lesson: benchmark the work product the next stage consumes. Tokens per second says whether a model is fast, and an agent benchmark says whether it uses tools. Neither says whether its extracted knowledge will make tomorrow's agent better.
Method notes: frozen production snapshot; 15 retrieval tasks, 3 repeats per ingest lane; answering agent pinned to Qwen3 30B and judge to deepseek-v4-flash. Aggregate differences were treated as tied within the 95% confidence interval. Content pass includes semantically perfect answers rejected only by the known citation-lineage artefact.
LLM Personalities: Part 1: The briefing mattered more than the hire · Part 2: One knowledge base, two AI jobs · Part 3: The evaluation harness was wrong too
Related reading: The story of making local agents usable · Hardware, engines, and benchmark results