← Engineering Insights

Why Enterprise RAG Pilots Fail After the Demo (And How to Fix Retrieval Latency)

Proof-of-concept retrieval looks strong on curated files. These are the structural failures—and the latency fixes—that appear on production Microsoft 365 corpora.

8 min read 2026-08-15 RAG Evaluation Azure

The demo illusion

Fixed 512-token windows, cosine similarity, and a generic system prompt can produce an impressive afternoon demo. Accuracy collapses when the same defaults index thousands of live SharePoint documents, PDF manuals, and nested repositories.

The four production failure modes

Arbitrary chunk boundary splitting

Token windows cut tables, conditionals, and safety steps in half. Use layout-aware parsing so section headers and full table cells stay intact.

Metadata stripping and temporal blindness

Without effective date, version, and audience tags, a 2024 policy chunk ranks beside the 2026 replacement. Inject metadata into chunk headers at index time.

Pure vector search over-reliance

Embeddings miss exact identifiers such as ERR_CONN_RESET. Fuse dense search with PostgreSQL tsvector using Reciprocal Rank Fusion.

Missing evaluation gates

Anecdotal prompt edits silently break other queries. Run LLM-judge suites for faithfulness and citation recall in CI before merge.

How to fix retrieval latency after the demo

Production latency is usually a retrieval problem, not a model problem. Hybrid search over an unfiltered corpus, oversized top-k, and synchronous embedding of every query add seconds users will not wait for.

  • Pre-filter before rank: Apply Entra ID / ACL predicates and recency windows before vector search so the ANN index is not scanning unauthorized rows.
  • Cap candidate fan-out: Retrieve a small dense set and a small sparse set, then RRF. Expanding top-k to “be safe” is the fastest way to blow p95.
  • Cache embeddings and rerank selectively: Embed queries once per session key; reserve cross-encoders for the final 8–12 chunks, not the full recall set.
  • Budget the path: Measure ingest-to-query p50/p95 separately from generation. If retrieval exceeds ~400ms, fix indexes and filters before changing models.
query
  → ACL + recency filter
  → dense k=20 + sparse k=20
  → RRF merge
  → optional rerank k=10
  → evidence threshold
  → generate or abstain

Production readiness checklist

  • Parsers keep headers and full table cells.
  • Every chunk stores source URI, last modified date, and permission tags.
  • Hybrid retrieval is benchmarked on real search logs, including latency.
  • Low-confidence queries abstain instead of generating.
  • Evaluation suites run in CI before prompt or index changes merge.

Is the RAG pilot stalling in production?

Audit retrieval performance, chunking, and governance boundaries with a senior engineer.

Request a RAG Architecture Review