Problem Statement

Users began reporting that the INESC TEC Research Assistant was "taking ages" to respond, especially when multiple colleagues were using it simultaneously. A quick look at our monitoring confirmed it: median latency at 2 concurrent users was 22 seconds, and some requests at 4 concurrent users exceeded 42 seconds.

The system was running a 14B-parameter model via vLLM on an NVIDIA L40S (48GB VRAM, FP8 quantization). On paper, this hardware should handle a model of this size with ease. Something was wrong at the configuration and application layer, not the hardware.

Investigation

We started with a concurrency benchmark, sending realistic HR and research queries simultaneously. This gave us a clean baseline before touching anything:

Three distinct root causes emerged:

1. GPU driver mismatch. The GPU driver kernel module and the installed userspace runtime were on different versions, leaving the GPU in a degraded state: high memory usage reported but near-zero compute utilization. Restarting the system reloaded the correct driver state.

2. A stuck background rebuild process. A knowledge base rebuild job had completed its core work but entered an infinite loop in post-processing, consuming substantial CPU and memory while competing directly with the inference engine. We confirmed the rebuilt index was intact before terminating the process.

3. Inference engine configured too conservatively for concurrent load. The concurrent request limit was set far below our actual usage patterns. With multiple users sending requests that each require two model calls internally, the queue filled quickly. There was also no request interleaving configured, so a long prompt from one user stalled token generation for all other in-flight requests.

Additionally, two intermediate pipeline steps were running in extended reasoning mode - generating full internal reasoning chains for tasks that only need brief yes/no answers.

Solution

Fix 1: Resolved environment issues. A system restart corrected the GPU driver state. The stuck background rebuild was terminated after confirming search results remained correct.

Fix 2: Simplified reasoning on intermediate pipeline steps. Two internal steps were configured for deep chain-of-thought reasoning despite only needing simple yes/no answers. Disabling extended reasoning on these utility calls reduced their individual latency by over 65%.

Fix 3: Increased inference engine concurrency. We raised the concurrent request limit and enabled two scheduling optimizations: request interleaving (long prompts no longer block other requests) and prefix caching (shared system prompt state reused across requests, ~80% cache hit rate in steady state).

Results

Concurrent users	Before p50	Before p95	After p50	After p95	Change
1	10.9s	11.0s	12.9s	13.5s	+2s (interleaving overhead)
2	22.4s	40.8s	13.9s	43.1s	-38% p50
4	17.9s	42.3s	12.0s	22.9s	-33% p50, -46% p95
8	38.2s	58.4s	9.2s	30.0s	-76% p50, -49% p95

8 concurrent users now respond faster (9.2s median) than a single user did before the fix (10.9s). This is the signature of effective batching: as concurrency increases, the inference engine can overlap work across requests and cache hit rates improve.

Single-user latency increased by ~2s - the expected trade-off of request interleaving on sequential workloads, which dramatically benefits concurrent ones.

Open Problems

p95 variance under high concurrency remains elevated. Cold-start rounds before the shared prompt cache warms up drive p95 significantly higher than steady-state. Pre-warming the cache at service startup is the planned fix.

Pipeline call depth is the latency floor. The irreducible minimum latency is set by the number of sequential model calls per request, not hardware. Reducing call depth is the primary lever for future improvement.

Running multiple model sizes simultaneously is memory-constrained on a single GPU. A fast-path for simpler queries using a lighter model would improve throughput but requires additional hardware or a dynamic model-swapping approach.

References

Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

vLLM documentation, Chunked Prefill and Automatic Prefix Caching.