Legacy Knowledge Copilot

The Situation

For mid-market engineering and operations teams, critical knowledge is rarely located in a single, well-structured directory. In this engagement, the client’s operational guidelines, server modernization runbooks, and software API documentation were scattered across three SharePoint sites, a Confluence instance, and isolated local network folders.

As a result, engineers spent an average of 45 minutes finding accurate technical procedures during active systems outages. In addition, onboarding new technical staff took up to six weeks because of knowledge discovery friction.

The Constraints

The client's security officers and data stewards established three non-negotiable boundaries before any code was written:

• Strict Tenant Containment: Proprietary designs, codebase notes, and operational logs could not leave the client's Azure boundary or be exposed to public model training loops.
• Zero-Hallucination Guardrails: The AI system must only respond using information retrieved from verified source text. If a query could not be answered from the document chunks, the system had to state "I do not know based on the available documentation."
• Access-Level Security: Search results and generated answers had to enforce SharePoint security groups, ensuring a user could never retrieve information they did not have explicit permissions to view in the source systems.

The Approach

We bypassed standard character-count text splitting—which frequently breaks code blocks and tables—to deploy a custom, semantic, layout-aware parser in Python. The custom parser groups paragraphs based on document heading structures, preserving parent headers and hierarchical outlines as metadata for every chunk.

For retrieval, we configured a hybrid search pipeline combining dense vector embeddings (using Azure OpenAI's text-embedding-3-large) with BM25 keyword matching. To prevent model hallucination, we enforced a strict retrieval similarity threshold and structured the system prompt to require inline citation anchors (e.g., [DocumentName.pdf, Page 12]) for every factual claim in the response.

Architecture & Technical Design

Below is the system architecture showing how documents are securely ingested and how user queries are processed and verified within the client's Azure network boundary.

This layout ensures that no unauthenticated user can access restricted documentation, and every single LLM call is tracked for cost, latency, and token footprint.

Delivery Model

We executed this project in a 4-week fixed-scope sprint. We worked directly as senior partners, with daily check-ins on Slack and weekly iterations demonstrating working code directly in the client's Azure staging environment. Because we bypass administrative layers and junior engineers, we moved from blueprint draft to a fully functional pilot in 20 engineering days.

The Results

• 75% Retrieval Speed Increase: Average information discovery times fell from 45 minutes to under 10 seconds.
• 100% Citation Coverage: Response filters intercept answers lacking clear source page links, preventing any hallucinated output from reaching the team.
• Zero Boundary Leaks: All search queries, embeddings, and chat histories remain strictly contained within the client’s Azure subscription.

Lessons Learned

Ingestion is 90% of RAG: Standard recursive text splitting degrades the usability of tables and code blocks. Layout-aware chunking (e.g. keeping table rows linked to table headers) is essential for retrieval correctness.

System prompts aren't enough: Enforcing strict retrieval distance ceilings (relevance thresholds) is the only way to reliably force the model to output "I do not know" instead of generating convincing hallucinations when documentation is missing.