Problem Statement

Following the concurrency improvements from SRL-001, a broader benchmark with more diverse query patterns revealed a separate critical problem. Most queries responded in 10-13 seconds, but a specific category of query was consistently taking over 95 seconds - more than 8x slower than everything else. The worst offender was a common, natural question about employee benefits that any new team member would ask.

The target was SOTA performance for a local 14B model on a single GPU: p50 under 5 seconds.

Investigation

A targeted benchmark isolated queries with broad thematic vocabulary as consistent worst-case offenders. Profiling the search pipeline showed a specific retrieval step taking over 90 seconds on those queries alone.

Keyword-based search scales in proportion to the number of query tokens: more tokens means more scoring work across the full document index. The problematic queries were arriving at the keyword search component with far more tokens than intended - over 100 in some cases.

Root cause: a query expansion function designed for semantic (dense) vector search was incorrectly applied to keyword-based (sparse) retrieval as well. The expansion adds synonyms and bilingual translations to improve recall - the right behaviour for semantic search. Applied to keyword search, a short query expanded into 90+ terms through cascading synonym chains, roughly doubling scoring work with each expansion and producing the 95-second spike.

All Fixes Applied

1. Separated query expansion paths for semantic and keyword search. The two retrieval strategies have fundamentally different requirements. Separated, the worst-case query dropped from 95,000ms to ~400ms.
2. Reduced prompt verbosity in model instructions. Internal capability descriptions condensed by 44%, saving ~3,500 input tokens per model call.
3. Fixed prompt structure to restore prefix cache hits. A dynamic element at the start of the static system prompt invalidated the inference engine's cache daily. Moving it to the end restored ~80% cache hit rates.
4. Eliminated first-request cold-start delay. A large search index was loading on demand at first query, causing a 25-second delay after each deployment. It now preloads at service startup.
5. Optimized ranking sort for large document sets. Replaced a general-purpose sort over 841,000 documents with a vectorized equivalent (~10x speedup for this step).
6. Removed automatic retry model calls on low-confidence results. Every low-confidence result triggered two additional model calls, adding ~3,500ms per retrieval step.
7. Fixed a logic error including unnecessary context in every response. A condition bug caused a large block of context to be included in every synthesis call even when it had no effect.
8. Added inference engine warm-up and cache quantization at startup. Cache quantization doubled available capacity with no hardware change.

Results

Lessons

• Retrieval strategies designed for semantic search can be actively harmful when applied to keyword-based search. The two approaches have fundamentally different scaling properties - what improves recall in one can degrade performance in the other by orders of magnitude.
• Caching only works when the cached content is truly static. Even a single dynamic element at the start of otherwise-static content silently breaks caching for every request.
• Internal pipeline steps that trigger additional model calls compound invisibly. Users experience the sum of every hidden call, and those costs multiply with each retrieval step. Count your model calls.