Enterprise RAG Evaluation for Microsoft 365: A Practical Production Readiness Framework

A demo copilot that answers three scripted questions in a sprint review proves only that a model can produce plausible text from retrieved context. It does not prove that the system will preserve Microsoft Entra ID permission boundaries, retrieve the current Azure runbook at 2:00 AM, or avoid regressions after a change to chunking, embeddings, ranking, or prompts.

Deploying internal RAG copilots across Microsoft 365, SharePoint, and Azure requires an explicit, repeatable evaluation harness. This framework provides an operational test methodology, core metrics, failure cases, and CI/CD gate design for verifying an enterprise RAG system before production rollout.

Why “It Answered Correctly Once” Fails

Ad-hoc chat testing creates a false sense of reliability. In enterprise copilots connected to dynamic knowledge bases—such as SharePoint document libraries, Teams archives, Azure Repos, and ticket systems—failures are rarely obvious total outages. More often, they are silent authorization failures, retrieval drift, and ungrounded answers.

• Anecdotal bias: Hand-tested prompts tend to avoid edge cases, unusual terminology, ambiguous requests, and multi-hop questions.
• Context bleed: Testing as an administrator can hide authorization-filtering bugs and permission leaks.
• Retrieval-generation decoupling: A plausible answer does not prove the right evidence was retrieved. The retriever may have returned irrelevant chunks while the model guessed correctly from pretraining or general knowledge.
• Silent regressions: A new embedding model, chunk size, token budget, ranker, or system prompt can improve a few showcase queries while degrading important production cases.
• Stale evidence: Archived runbooks and superseded policies can remain indexed and appear credible unless versioning and authority signals are tested explicitly.

Build a 40-Question Golden Set

Evaluation does not require thousands of labeled examples to begin. A curated suite of 30 to 50 deterministic cases, derived from real SharePoint sites, Teams archives, Azure Repos runbooks, and ticket logs, can expose the most important production regressions.

Each test case should include the query, a non-production test principal or Entra ID claims fixture, expected behavior, expected authorized evidence sources, explicitly forbidden sources, and the expected outcome category: answer, abstain, or deny.

1. Grounded synthesis questions

Use multi-sentence questions that require synthesis across two to four distinct, authorized sources. For example:

"How do our incident escalation tiers differ between Tier 1 Azure outages and third-party SaaS downtime?"

The answer should combine the relevant evidence without introducing unsupported policy details. Each material claim should link to a source that substantiates it.

2. Exact identifier and configuration lookups

Include queries involving Azure subscription IDs, error codes, environment variables, service names, policy identifiers, or repository-specific configuration values. Pure vector search can perform poorly on exact strings, so these cases validate lexical retrieval, hybrid search, and Reciprocal Rank Fusion (RRF).

3. Unanswerable and out-of-scope prompts

Test questions for which no authorized evidence exists, such as:

"What is our company policy on personal quantum computing rigs?"

The expected result is not a generic corporate-policy answer. The system should state that the authorized documentation does not contain sufficient information to answer.

4. Cross-tenant and ACL-restricted queries

Run restricted-content questions under non-privileged test identities. Examples include executive compensation, legal audits, incident postmortems, M&A strategy, and HR policies. The test must confirm that security trimming excludes unauthorized sources before ranking, prompt construction, tracing, citations, or generation.

5. Stale and contradictory document tests

Add cases where the corpus contains an obsolete procedure and a current authoritative replacement. For example, an archived 2024 VPN guide may conflict with a current 2026 Entra Private Access guide. The system should retrieve the current document and avoid mixing obsolete configuration steps into a response.

Measure the Pipeline, Not Just Answers

Do not use generic text-overlap metrics such as BLEU or ROUGE as a primary production-quality signal for an enterprise RAG system. Measure isolated retrieval, generation, citation, and governance behavior.

Authorization isolation

Authorization isolation is a binary security control: an unauthorized source must never be retrieved, included in context, exposed in a trace, cited, or reflected in generated text. Any unauthorized content exposure is a release-blocking vulnerability.

Evidence recall@K

Evidence recall asks whether the retrieval system surfaced at least one expected, authorized source for a test case in its top-K candidates. Low recall typically points to chunking boundaries, incomplete ingestion, poor embedding alignment, stale indexes, insufficient lexical retrieval, or weak hybrid-search weights.

Ranking quality and precision@K

Precision evaluates whether relevant sources rank near the top of the result list rather than being buried below distractors. This matters because context windows, rerankers, and generation behavior give disproportionate influence to the first few chunks.

Track performance by query type. Exact IDs, multi-source policy questions, and short ambiguous prompts often behave differently and should not be hidden inside a single aggregate score.

Claim groundedness

Groundedness measures whether factual, material claims in a response are supported by retrieved evidence. A statement without direct evidence is a hallucination even if it sounds reasonable or happens to be true.

For high-risk internal use cases, initially target at least 95% grounded material claims, then tune the threshold according to the system's business impact and permitted failure modes.

Citation correctness and abstention

Citation quality contains two separate checks:

• Citation validity: Each citation refers to a retrieved, authorized source.
• Citation entailment: The cited passage actually supports the claim attached to it.

A system can pass citation validity while still providing misleading citations. A retrieved document ID proves only that the source was present in context; it does not prove that the cited text substantiates the sentence.

Abstention must also be evaluated. When the authorized evidence is missing, conflicting, stale, or insufficient, the system should decline clearly rather than create a plausible answer. For example:

"The authorized documentation available to this copilot does not contain enough evidence to answer that question."

Failure Cases to Test Before Rollout

Put Evaluation in CI/CD

Evaluation should not be a manual exercise conducted before a quarterly release. Treat it as a release gate in Azure DevOps or GitHub Actions. Changes to chunking, ingestion, embedding models, retrieval configuration, ranking thresholds, system prompts, access-control code, or model providers should trigger the evaluation suite.

# .github/workflows/rag-eval-gate.yml
name: RAG Production Readiness Gate

on:
  pull_request:
    paths:
      - 'src/retrieval/**'
      - 'src/prompts/**'
      - 'src/authorization/**'
      - 'config/search-index.json'
      - 'tests/golden-eval-set.json'

jobs:
  evaluate-rag:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -r requirements-eval.txt

      - name: Execute evaluation matrix
        env:
          AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
          EVAL_DATASET_PATH: './tests/golden-eval-set.json'
        run: |
          python -m eval.run_evaluation \
            --dataset $EVAL_DATASET_PATH \
            --baseline ./tests/eval-baseline.json \
            --min-faithfulness 0.95 \
            --min-recall 0.85 \
            --min-precision 0.80 \
            --strict-acl-check true \
            --fail-on-critical-case true \
            --max-category-regression 0.02 \
            --output-report ./eval-report.json

      - name: Publish evaluation artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: rag-evaluation-report
          path: ./eval-report.json

A useful release policy is simple:

• Fail immediately for unauthorized retrieval or any restricted-content exposure.
• Fail immediately when an unanswerable or restricted case produces a substantive answer.
• Fail when citation validity or citation entailment fails for a high-risk response.
• Flag or block material regressions against the approved baseline, even if the aggregate average still passes.
• Publish a report that includes per-case sources, ranking positions, citations, abstention status, latency, model version, retrieval configuration, and index version.

Validate the Real Authorization Path

Do not duplicate ACL logic inside an evaluation script. The evaluator should call the same authorization policy service used by the production retrieval path. Otherwise, a test can pass even though the evaluator and production service interpret permissions differently.

The following is illustrative pseudocode:

from typing import Dict

def evaluate_test_case(item: Dict, model_output: Dict, authorization_service) -> Dict:
    principal = item["test_principal"]
    claims = item["user_claims"]

    # Validate the actual policy decision for every retrieved source.
    for chunk in model_output.get("retrieved_chunks", []):
        if not authorization_service.can_read(
            principal=principal,
            resource=chunk["source_resource"],
            claims=claims,
        ):
            return {
                "passed": False,
                "reason": f"ACL leak: unauthorized chunk {chunk['id']} was retrieved"
            }

    # Unanswerable and unauthorized cases must decline safely.
    if item["expected_behavior"] in {"ABSTAIN", "DENY"}:
        if model_output.get("abstained"):
            return {"passed": True, "score": 1.0}
        return {
            "passed": False,
            "reason": "System generated an answer when it should have abstained or denied."
        }

    cited_ids = set(model_output.get("citations", []))
    retrieved_ids = {chunk["id"] for chunk in model_output.get("retrieved_chunks", [])}

    # Citation validity: no invented or non-retrieved citation IDs.
    if not cited_ids.issubset(retrieved_ids):
        return {
            "passed": False,
            "reason": "Citation validity failure: an answer cited a source not retrieved."
        }

    # Citation entailment should be evaluated separately using claim-to-evidence checking.
    return {
        "passed": True,
        "score": model_output.get("faithfulness_score", 0.0),
    }

This pseudocode only validates that citations identify retrieved sources. A production harness must additionally evaluate whether each cited source actually entails the claim it is presented as supporting.

Evaluation Must Be Versioned

A golden dataset is useful only when the team can reproduce and compare results over time. Version each test case and preserve the configuration that produced a result.

Store the following with every case:

• Query text and the test principal or Entra ID claims fixture.
• Expected authorized sources and explicitly forbidden sources.
• Expected behavior: answer, abstain, or deny.
• Expected citation sources rather than brittle chunk IDs where possible.
• Dataset version, document version, source-system identifier, and index timestamp.
• Embedding model, chunking strategy, overlap, hybrid-search weights, top-K, reranker, prompt version, and model version.
• Evaluation scores, trace identifiers, baseline release identifier, and remediation notes for known failures.

This transforms evaluation from a one-time quality exercise into an auditable engineering practice. It also makes it possible to understand whether a regression came from source document changes, ingestion, retrieval, authorization, prompt behavior, or the generation model.

For more details on production permissions and retrieval pipelines, see our guide on Designing Citation-Grounded RAG for Microsoft 365.

Hardening Before Deployment

A copilot should not move to production because a stakeholder liked a demo. It should move when the team can reproduce its behavior, inspect its retrieved evidence, verify that authorization was enforced before retrieval, and demonstrate that unsupported questions lead to safe abstention rather than plausible invention.

Before exposing an internal search copilot to enterprise users, run a focused evaluation and hardening sprint:

1. Build a representative 30- to 50-question golden dataset.
2. Establish baseline behavior across retrieval, grounding, citation, abstention, and authorization tests.
3. Add automated regression gates to the CI/CD pipeline.
4. Test high-risk permission boundaries with controlled non-production identities.
5. Document release criteria, known failure modes, remediation ownership, and re-evaluation triggers.