Right tool for the job: why my chat engine lost the reranking race

By Steve Miles, software & hardware engineer in Perth, Australia.

After months of tuning, llama.cpp's SYCL backend earned the chat slot on my Intel Arc Pro B60. It beat the official vendor stack on the single-user agent loop, its flash-attention path unlocked prefix caching, and quantised models left VRAM to spare. The B60 story covers that journey.

So when the knowledge base needed a reranker, the choice looked obvious: same engine, same card, one more container. That assumption cost the system a 10× latency penalty for weeks before I measured it.

Same model, two engines

A reranker is a cross-encoder: it takes a query and a candidate document and returns one relevance score. My retrieval pipeline calls it on every search, scoring 25 to 50 pairs per query. It sits directly on the user's critical path, unlike ingest work that can run overnight.

The test was as controlled as it gets. One model, bge-reranker-v2-m3, served two ways on the same card:

• llama.cpp SYCL, Q8 GGUF, the custom F16 build that wins my chat benchmarks
• Hugging Face Text Embeddings Inference (TEI) with Intel's XPU-IPEX runtime, fp16 safetensors

Five repetitions each, after warmup:

The engine that dominated chat lost reranking by an order of magnitude, and on long documents it did not lose so much as fail: the batch limit that is harmless for a single chat stream cannot hold multiple concurrent long pairs, and the server returns an error instead of a score.

The community expectation for TEI on Intel was a 2–3× improvement, a figure set against CUDA hardware. On Battlemage the measured gap was 7–9×. Published numbers from other people's stacks are hypotheses; the earlier engine lessons apply to rerankers too.

Why the winner changed

Nothing about this contradicts the chat result. The workloads reward different things.

A chat agent lives on decode speed, prefix-cache reuse, and fitting a quantised 26B model into 24 GB. llama.cpp is built around exactly that: GGUF quantisation, KV-cache management, one long-running conversation.

A reranker is the opposite shape: hundreds of tiny independent forward passes, no generation, no cache to reuse, throughput over a batch as the only number that matters. TEI runs the model at fp16 through IPEX with fused attention and proper batching, and the 278M-parameter model is small enough that quantisation buys nothing you need.

The cost of the switch was about 1.5 GB extra VRAM for the fp16 weights and runtime. For a 9× latency cut on the search path, that trade took no deliberation.

The same logic later killed a tempting consolidation: a ColBERT-style late-interaction model that could have unified retrieval and reranking benchmarked at 2.7× slower than the cross-encoder and needed 21× the vector storage. Elegant architecture, wrong tool.

The migration trap: scores changed meaning

Swapping the endpoint was the easy part. The subtle part nearly shipped broken:

llama.cpp returns raw logits; TEI returns sigmoid probabilities. A strong match scores +5 to +8 in logit space but 0.99-something in probability space. My service graded confidence with logit-scale thresholds ("high" above 2). Pointed at TEI without recalibration, "high" could never fire again, and every search result would have been labelled medium confidence forever, with no error anywhere.

The fix is arithmetic, not code: the old logit thresholds of 0 and 2 map to sigmoid 0.5 and 0.88. But the lesson generalises: when you change a serving engine, the numbers that come back may change meaning while the API shape still works. Recalibrate anything downstream that interprets scores.

The wire formats also differ (different request field names, different response shapes), so the client now sends both field variants and parses both response shapes. One client, either engine, which made the rollback path trivial.

One more production lesson

Under real traffic, the TEI container's VRAM climbed from 1.4 GB to over 6 GB in an hour: a cache-release leak in the XPU path. The fix was two lines, calling the runtime's cache-release after each rerank batch, at a cost of about 1ms per call. VRAM has held flat since. New engine, new failure modes; watch the resource graphs before decommissioning the old one. The llama.cpp rerank container stayed running as a fallback for a week before it was retired.

The takeaway

"Which engine is best on this GPU" is an incomplete question. On the same B60, on the same day:

• Chat and agents: llama.cpp SYCL, for cache reuse, quantisation, and VRAM fit.
• Reranking: TEI XPU-IPEX, for 7–9× the throughput and long-document robustness.
• Embeddings: llama.cpp again, where its batch handling was never the bottleneck.

Benchmark per model class, not per card. The engine that wins your headline workload can be the slowest option for the workload sitting next to it, and the only way to find out costs one afternoon of curl and a stopwatch.

Method notes: bge-reranker-v2-m3 served as Q8 GGUF on llama.cpp SYCL and as fp16 safetensors on TEI XPU-IPEX, same Intel Arc Pro B60, 5 repetitions per test after warmup. Latency figures are end-to-end HTTP round trips. The production cutover retained the old endpoint for evaluation-baseline comparability, since confidence labels shift with the score scale.

Related reading: Intel Arc Pro B60 from disappointing to practical local AI Agents · Hardware, engines, and benchmark results · LLM Personalities series