Problem Statement

Four distinct classes of response accuracy failure were identified through user reports and internal monitoring over a two-week period. Unlike latency issues, these failures were silent: the system returned a confident-looking answer that was factually wrong for the user's actual question.

Investigation

Failure 1: Cross-topic source contamination. The system builds a rolling session summary and injects it into the current query context to maintain continuity. The bug: a presearch shortcut designed to inject domain-specific data was reading this accumulated session summary instead of the user's actual current message. After a topic switch, the old topic was still in the summary, causing the presearch to select documents for the wrong topic.

Failure 2: Concluded resources described as active. The filter that excludes concluded or closed resources when a user asks for active work checked for English and Brazilian Portuguese terms only. Our knowledge base uses European Portuguese in some records, which passed through undetected.

Failure 3: Domain routing failure after topic switch. The streaming response path was missing a block of context-handling logic that exists in the non-streaming path. When a user switches from a project discussion to an HR or policy question, the system should route the search to the HR knowledge domain. In the streaming path this was absent, so it searched the wrong domain.

Failure 4: Team membership returning low-confidence results. Membership queries rely on vector search finding a specific structured section of a project record. Without the project name in the current message, the retrieval ranked general pages higher than the structured project file, returning a low-confidence answer from the wrong documents.

Fixes Applied

Fix 1: The presearch condition now evaluates only the user's current message, not the accumulated session summary. Session summary is still used for general context injection but no longer controls domain-specific source selection.

Fix 2: The closed-resource filter was extended to recognise European Portuguese vocabulary, alongside the existing English and Brazilian Portuguese terms.

Fix 3: Topic-switch detection and HR domain routing were added to the streaming response path, bringing it to parity with the non-streaming path.

Fix 4: A deterministic presearch step was added for team membership queries: when the query matches membership vocabulary and a known project is in the conversation context, the system reads the structured project record directly rather than relying on vector search. This bypasses retrieval uncertainty entirely for this query class.

Results

All four open user reports for these failure classes were closed as resolved.

Lessons

• Streaming and non-streaming code paths diverge silently. Any logic added to one must be explicitly applied to the other. We now treat them as requiring parallel updates by default.
• Session context that helps continuity in normal conversation becomes a liability when the topic changes. Separate what controls topic routing from what provides conversational continuity - they should never be the same variable.
• For highly structured, factual queries, vector retrieval introduces unnecessary uncertainty. A deterministic lookup against a known structured record beats retrieval when the data location is predictable.
• Knowledge base coverage gaps - missing vocabulary variants, outdated records - produce confident wrong answers rather than visible errors. Regular coverage audits and broader vocabulary matching are part of the solution.