Enterprise RAG Knowledge Copilot (Backend Architecture)

The Situation

For engineering and operations teams, critical knowledge is rarely located in a single, well-structured directory. In this engagement, operational guidelines, server modernization runbooks, and software API documentation were scattered across SharePoint sites, a Confluence instance, and internal repositories.

As a result, engineers spent an average of 45 minutes finding accurate technical procedures during active systems outages. In addition, onboarding new technical staff took up to six weeks because of knowledge discovery friction.

The Constraints

Security and platform owners established non-negotiable boundaries before any code was written:

• JWT/RBAC Access Control: Every query path authenticated and authorized so users could only retrieve documentation they were permitted to see.
• Prompt-Injection Guardrails: Untrusted user input could not override system instructions, exfiltrate secrets, or bypass retrieval policy.
• Citation-First Guardrails: The system must only respond using verified retrieved source text. If a query could not be answered from document chunks, it had to state "I do not know based on the available documentation."
• 100% Citation Coverage: Every factual claim required an inline source citation before the answer could reach the user.

The Approach

We deployed a layout-aware ingestion path in Python that preserves heading hierarchy, tables, and code blocks as structured chunk metadata — avoiding brittle character-count splits that destroy retrieval quality.

For retrieval, we built a FastAPI service backed by PostgreSQL/pgvector hybrid vector and keyword search, fused with Reciprocal Rank Fusion (RRF) and semantic query routing. Generation was citation-bound, with prompt-injection defenses at the gateway. An automated LangSmith LLM-judge evaluation workflow scored answer faithfulness and citation coverage so regressions could not ship unnoticed.

Architecture & Technical Design

Below is the system architecture showing how documents are ingested into PostgreSQL/pgvector and how authenticated queries flow through FastAPI retrieval, guardrails, citation-bound generation, and LangSmith LLM-judge evaluation.

This layout keeps retrieval and generation behind JWT/RBAC controls, fuses hybrid candidates with RRF, and uses LangSmith LLM-judge evaluation to verify citation coverage, faithfulness, and regression risk before changes reach production.

Delivery Model

We executed this project in a 4-week fixed-scope sprint. We worked directly as senior partners, with daily check-ins on Slack and weekly iterations demonstrating working code in the client's staging environment. Because we bypass administrative layers and junior engineers, we moved from blueprint draft to a fully functional pilot in 20 engineering days.

The Results

• 75% Retrieval Speed Increase: Average information discovery times fell from 45 minutes to under 10 seconds.
• 100% Citation Coverage: Response filters intercept answers lacking clear source links, preventing hallucinated output from reaching the team.
• Controlled Access Boundary: Queries, embeddings, and chat histories remain inside the approved deployment boundary with JWT/RBAC enforcement on every request.

Lessons Learned

Ingestion is 90% of RAG: Standard recursive text splitting degrades the usability of tables and code blocks. Layout-aware chunking (e.g. keeping table rows linked to table headers) is essential for retrieval correctness.

System prompts aren't enough: Hybrid retrieval with RRF, relevance thresholds, prompt-injection defenses, and automated LLM-judge evaluation are what reliably force "I do not know" instead of convincing hallucinations when documentation is missing.