Assistant end-to-end evaluation — developer guide

Purpose

This document describes the design and internals of the assistant evaluation harness (RAG goal R1b). The harness measures the composed InAppAssistanceService::respond() over a per-module dataset in two modes:

• Level 1 (deterministic, built now, CI regression gate): runs the real respond() with a scripted router (fake completion provider) over the real documentation, application-content, and Graph retrieval paths. It measures composition plumbing — scope gating, citation assembly, clarification, abstention, and output validation — not the LLM's surface-routing accuracy.
• Level 2 (live, defined now, built later, opt-in): runs the same dataset against a live LLM to measure actual surface-routing, citation precision, and answer quality. Non-deterministic; outside CI and specified but unimplemented.

See also: ASSISTANT_SCOPE.md (module-scoped documentation retrieval that L1 verifies), DOCUMENTATION_EVALUATION_DEVELOPER.md (the mirror evaluation harness for documentation retrieval only, which L1 reuses).

Capabilities

Deferred (Level 2):

The dataset contract mirrors DocumentationEvaluationDataset: JSON with exact-key validation, bounded sizes, and synthetic-only data classification. The reading of the file, the typed accessors and the statistics are shared code (Modules\AI\Services\Evaluation\*, see DOCUMENTATION_EVALUATION_DEVELOPER.md); the assistant dataset keeps its own optional-key helpers.

Internal Flow

Level 1 (deterministic, built)

1. The regression gate test loads a dataset JSON from Modules/{Module}/docs/rag/evaluations/assistant-{slug}.json into AssistantEvaluationDataset.
2. For each case, AssistantEvaluationService::evaluate():
   • Resolves the case's AssistantAccessContext (profile fixed to InAppAssistance, module/locale/permissions from the case).
   • Constructs InAppAssistanceService with:
     • Real documentation_retrieval (reusing R0 FakeDocumentationSearch fixture).
     • Fake application-content and Graph providers (fixture stubs with authored evidence/ambiguity per case).
     • Scripted completion closure from ScriptedAssistantRunner — given the case's expected_surface, invokes the matching tool with scripted arguments over the fake provider (which sets citation metadata), then returns a scripted answer string.
   • Calls the real respond($query, $context) method.
   • The scripted completion, per expected_surface, either:
     • Invokes application_content_search tool and scripted args → fake provider sets ApplicationContentCitationMapper state → returns scripted answer.
     • Invokes graph_* tool with scripted args → fake Graph provider → scripted answer.
     • Invokes nothing, so deterministic clarification/abstention paths in respond() fire → localized clarification/refusal output.
     • For documentation surface, uses the pre-retrieved documentation context.
   • respond() then runs post-processing (citation merge, abstention check, output validation, persistence) unchanged.
3. Scores the case per module:
   • citation_assembly — returned Message metadata citations contain the expected set.
   • clarification_trigger_accuracy — clarification output iff expect_clarification.
   • abstention_accuracy — abstention output iff a surface was attempted with no evidence (refuse cases).
   • output_valid — scripted output passed mandatory validation.
   • proposal_accuracy — among the cases that carry expected_proposals, the share whose message holds exactly that many proposals in metadata.proposals.
   • pending_report_accuracy — among the messages that carry a proposal, the share whose text does not claim that a change was made (AssistanceOutputPolicy::claimsChange(), English and Italian).
   • unavailable_rate — cases where respond() threw or returned unexpectedly (fail-closed).
4. Aggregates metrics per module and locale. Slices recompute each metric by module, locale, and case tag (deterministic ksort).
5. Outputs a report JSON containing only aggregate floats and slugged slice keys — never queries, citations, or record content.

Level 2 (live, defined, deferred)

When built, will run the same dataset over the real LLM (not a scripted completion) and measure actual surface-routing accuracy, citation precision across sources, real clarification/refusal behavior, and answer grounding against retrieved evidence. The future ai:evaluate-assistant --live command will switch to Level-2 mode. Level 2 is not yet implemented; this is a specification of the intended interface for a later phase.

How To Use — Add a Report Card for a New Module

1. Author the dataset: Create Modules/{Module}/docs/rag/evaluations/assistant-{slug}.json.

   Dataset structure (JSON):

   • version — schema version (string, e.g. "1").
   • corpus_revision — commit/version of the test fixtures (string).
   • module — lowercase module name (string, e.g. "cms").
   • data_classification — restricted to "synthetic" (string).
   • cases — bounded array (max 1000 entries). Each case:
     • id — unique case slug (string, kebab-case).
     • query — the question asked (string).
     • locale — language code (string, e.g. "en").
     • module_key — expected module scope (string, or null for generic/no-module case).
     • expected_surface — one of "documentation", "application_content", "graph", "clarify", "refuse" (string).
     • expected_citations — bounded list of safe citation labels/references (array of strings; empty for clarify/refuse).
     • expect_clarification — true iff expected_surface == "clarify" (boolean).
     • expect_refusal — true iff expected_surface == "refuse" (boolean).
     • slices — lowercase slug tags for breakdowns (array of strings, e.g. ["publishing", "single_hop"]).
     • page — optional. The context.page the case's client sends with the message: an object with a resource (module/entity) and, for proposal cases, a proposable list. At most 6000 bytes. The runner puts it in the request exactly as a client would.
     • expected_proposals — optional integer from 0 to 3: how many proposals the message should carry. Needs a page.

   Validation invariants:

   • clarify/refuse cases carry no expected citations.
   • expected_surface and boolean flags must agree (clarify → expect_clarification=true, etc.).
   • graph and application_content surfaces are valid only where the module scope could offer them (determined by case module_key and server configuration).
   • Slug patterns match ^[a-z0-9][a-z0-9_-]{0,63}$ (lowercase, digits, underscores, hyphens only; no colons).
   • page and expected_proposals are the only optional keys; any other key outside the list above is refused. A dataset without them reports proposal_accuracy and pending_report_accuracy as 0.0, like any metric with nothing to measure.

   Example excerpt:

   {
     "version": "1",
     "corpus_revision": "main:a1b2c3d",
     "module": "cms",
     "data_classification": "synthetic",
     "cases": [
       {
         "id": "cms-doc-content-search",
         "query": "How do I search for pages by keyword?",
         "locale": "en",
         "module_key": "cms",
         "expected_surface": "documentation",
         "expected_citations": ["user:cms-search-guide"],
         "expect_clarification": false,
         "expect_refusal": false,
         "slices": ["search"]
       }
     ]
   }
   

2. Author fixture providers: Add fake application-content and Graph provider stubs under Modules/AI/tests/Stubs/Assistance/ (mirror Modules/AI/tests/Stubs/Documentation/CoreUserDocumentationCorpus).

   ScriptedAssistantFixtures::forCMS() (or per-module method) should:

   • Map each case query (via crc32) to a fake provider response with authored citations and ambiguity state.
   • For application_content surface: return ApplicationContentResult with results (safe text + metadata) and an optional clarificationRequired flag.
   • For graph surface: return GraphResult with relation/expansion results.
   • For clarify/refuse cases: return empty/ambiguous results so the guardrails fire.

3. Add the gate test: Extend Modules/AI/tests/Feature/Assistance/AssistantBaselineGateTest.php with a new test case for your module (mirror the existing CMS test).

   • Load the dataset via base_path('Modules/{Module}/docs/rag/evaluations/assistant-{slug}.json').
   • Run AssistantEvaluationService::evaluate() with the fixture providers.
   • Assert each Level-1 metric at >= 1.0 (hardcoded thresholds: citation_assembly, clarification_trigger_accuracy, abstention_accuracy, output_valid).
   • Assert unavailable_rate === 0.0.

4. Run the gate test:

   php artisan test Modules/AI/tests/Feature/Assistance/AssistantBaselineGateTest.php
   

5. Commit the dataset:

   git add Modules/{Module}/docs/rag/evaluations/assistant-{slug}.json
   git commit -m "test({module}): assistant evaluation baseline"
   

Proposals dataset

Modules/AI/docs/rag/evaluations/assistant-proposals.json evaluates the proposals of the adaptive assistance contract (Modules/AI/docs/rag/ASSISTANT_SCOPE.md explains where the page context comes from). Six synthetic cases, each with a page that declares a proposable target:

What Level 1 proves here. The scripted model is steered by the slices of the case (forged_hint, applied_claim, or none), so the harness proves the plumbing and the guard: a valid proposal reaches metadata.proposals, an undeclared target and an out-of-schema value never do, and an answer that claims a pending proposal was applied is replaced. It does not prove that a real model decides to propose when it should and abstains when it should not. That decision is a Level-2 measurement, and Level 2 is not built.

Result, 2026-10-06 (php artisan test Modules/AI/tests/Feature/Assistance/AssistantBaselineGateTest.php, scripted model, 6 cases):

Slices proposal, forged_hint, applied_claim, no_proposal and locales en and it hold proposal_accuracy 1.0. pending_report_accuracy is 1.0 on proposal, applied_claim, en and it, and 0.0 on forged_hint and no_proposal, which have no message with a proposal to measure. The other three metrics are 0.0 here because the dataset expects no citations, clarification or refusal.

Configuration

• ai.features.faq.max_documents — resolved top-K for documentation retrieval in evaluated cases (clamped 1–10). Tests set this to the dataset's intended K.
• ai.features.faq.policy_classification_version (the setting features.faq.policy_classification_version) — must match the fixture documents' policy_classification_version (default in-app-docs-v1).
• No new configuration surface for assistant evaluation. Scope and profile are server-owned and fixed per-request, never read from config or model output.

Permissions and Security

• Scope is server-owned: the module context comes from the case, never from user input or the assistant's own choice.
• No live LLM or Elasticsearch in Level 1: the real respond() path (scope resolution, guardrails, ACL-preserving tool authorization, output validation) runs; only the scripted completion provider is faked.
• Scripted router cannot widen surfaces: the completion can only invoke tools the resolved AssistantScope offers; it respects the same scope gating as production.
• Reports leak no payload: only aggregate metrics and slugged slice keys, never query text, citations, or record content.
• Data classification: restricted to synthetic to signal test-only data.
• Level 2 (when built) will run under the same server-owned profile/scope/ACL guarantees and remain opt-in, outside CI.

Performance and Limits

• Deterministic Level 1 only: no external services in tests.
• Per-module, dataset-driven: each module owns its evaluation dataset under Modules/{Module}/docs/rag/evaluations/assistant-*.json; datasets allow up to 1000 cases per module.
• Measures plumbing, not routing: the scripted completion choice means L1 proves "given surface X, the plumbing handled X correctly," not "the assistant chose X." Routing accuracy is a Level-2 metric (not yet built).
• Citation subset-coverage: after a scripted surface tool invocation, the message metadata citations are scored by subset-coverage — the returned citation labels must contain the expected set (no extra citations needed, but no missing ones).
• Fail-closed on retrieval failure: if respond() throws or returns unexpectedly, the case counts as unavailable with no impact on other metrics.

Errors and Troubleshooting

Related

• ASSISTANT_SCOPE.md — module-scoped documentation retrieval, server-owned scope resolution (AssistantScopeResolver), confirmed by L1.
• DOCUMENTATION_EVALUATION_DEVELOPER.md — mirror deterministic evaluation harness for documentation retrieval only (R0); L1 reuses its FakeDocumentationSearch fixture.
• Modules/AI/docs/rag/ASSISTANT_DATA_TOOLS_USER.md — end-user guide to application content and Graph tools.
• Design spec: docs/superpowers/specs/2026-08-29-assistant-end-to-end-evaluation-design.md.
• Forward contract: R0 (docs/superpowers/specs/2026-08-04-documentation-rag-evaluation-baseline-design.md) defined the assistant-level evaluation shape; this realizes it for Level 1.