2026-07-19 Phase 4 RAG, Evaluation, Safety, and Production Evidence Chain Record

This phase turns the Phase 3 queue’s “vertical review of RAG, evaluation, safety, and productionization” into an executable, fault-injectable, auditable evidence chain. The primary Agent standardized public/internal-data boundaries, terminology, versions, cross-links, and final verification. Specialist Agents separately reviewed RAG, LLMOps, Reranking, source links, document consistency, and code-bypass risks, with isolated scopes; the primary Agent reviewed and merged semantics.

Conclusion boundary

This phase deeply reviews offline teaching projects and high-priority main-path pages. It does not mean historical RAG, evaluation, safety, or production material is all page-by-page verified. Deterministic fixtures prove that local contracts can fail closed; they do not prove real vector retrieval, real LLMs, production identity, online telemetry, signed artifacts, or physical deletion.

1. RAG: from “has citations” to layer-by-layer recomputable evidence

• The offline citable-Q&A project now returns only stable status, answers rendered from verified claims, post-authorization citations, and a teaching trace_id. Candidates, filtering reasons, stage scores, internal versions, and failure details enter only privileged_audit.
• inspect —operator-view prevents mistaken disclosure of an internal teaching envelope; it does not authenticate an operator. Public access uses explicit public group rather than an empty group or magic ACL bypass.
• Fixture schema 2.0 includes 10 versioned documents, 8 query types, authorized revision, risk slices, critical cases, a private canary, and untrusted documents with pseudo-instructions. Validity uses half-open [effective_from, effective_to).
• Runtime queries are whitelist-projected and separated from offline oracles such as expected_*, forbidden IDs, and private canary. The runtime validator does not read the oracle to decide an answer.
• Retrieval, reranking, context selection, claims, citations, degraded/fallback state, public response, and teaching trace are recomputed from trusted inputs and stage results. Every citation for every claim must support that claim verbatim; duplicate citations and “one valid citation hides another invalid citation” are rejected.
• Evaluation output is fixed as rag-evaluation-report-v1, reporting case/slice/critical/status accuracy, retrieval/context/citation fact recall, and non-disclosure violations. Fixture/evidence use complete SHA-256. Fact-level counting and strict audit semantics are bound by offline-rag-harness-v3; normal run is PASS, fault injection is BLOCK.
• The main path adds an original layered-evidence Mermaid plus text alternative: ingestion, retrieval, context, generation, and system must not be collapsed into one “accuracy.”

2. LLMOps: bind offline score, release decision, and online window

• The Offline Release Gate evaluation artifact binds subject release, suite, dataset, rubric, grader, harness version, and complete SHA-256. Contract mismatch returns INCOMPARABLE, not a false improvement/regression.
• JSON input rejects duplicate keys; policy fields such as ratios, drop rate, safety ceiling, latency, and cost have explicit type/numeric limits.
• Online observation uses strict UTC RFC 3339 window; assignment/population revision; candidate/control arms; coverage, sample count, label age, quality, safety, latency, and cost; and binds candidate gate fingerprint plus control release. A candidate records gate_decided_at/promoted_at, with promoted_at <= window_start < window_end <= decision_as_of.
• Online judgment compares simultaneous controls in the same window, not historical offline baseline. audit —release-id requires observation; bulk audit requires every promoted candidate to have it but does not require online data from an offline-blocked candidate.
• Release IDs are globally unique. Floating refs such as latest, HEAD, branch/ref/remote forms fail closed. Time accepts only Z or +00:00; -00:00, no timezone, and non-UTC offsets are rejected.
• Automated FALLBACK cannot be triggered by an observation boolean. Online evidence must match the candidate gate’s bound fallback release ID, manifest SHA-256, and gate-evidence SHA-256 field by field. Missing evidence requires human review; nonempty mismatched evidence is rejected. External fallback artifacts are neither loaded nor signature-verified and remain Layer B.
• Binding unconditionally rejects nonempty mismatched fallback evidence. The manifest rejects self, known cycle, known blocked, or unpublished targets for automatic fallback. Policy moved to local-llmops-policy-v6, observation schema to v3, and all observations were rebound to candidate fingerprint. The short decision fingerprint is local correlation only, not a signature, provenance proof, or approval record.

3. Reranking and monitoring: expose hidden input and honest limits

• The Reranking fixture moves to schema 2 and adds authorization_revision. Its normalized SHA-256 covers query, tenant, groups, authorization version, as-of, candidate body/ACL/state/validity/source revision/rank/score, settings, qrels, and must-not-return. Evidence SHA-256 separately binds runtime overrides, faults, model version, and decision.
• Reranking output explicitly uses visibility=protected_audit. This label is neither public response nor real authorization; authorization/validity still precede scoring, with half-open end-time semantics.
• The Offline Monitoring Audit JSON loader rejects duplicate keys and corrects the teaching RED error-rate formula. Its Layer A verifies only local thresholds and incident classification; it does not bind full SLO population, exclusion rules, compliance window, data-source version, or change approval.

4. Cross-course consistency of facts, sources, and links

• Corrected MCP Roots: they coordinate suggested filesystem focus ranges from client to server; they are not authorization, ACL, sandbox, or least-privilege boundary. Client and route pages use the same wording.
• The frozen MCP reference layer received only narrow factual correction; later page-level curated/third-party and license decisions remain required.
• Knowledge Base examples are stated to have only a flat allow-list, not hierarchical permission, explicit deny, ABAC, or permission-source propagation. archived/tombstone is not physical deletion, retention compliance, or proof that old snapshots cannot revive.
• RAG troubleshooting uses protected inspect —operator-view, not public ask to imply internal-trace access. Future Provider sunset dates remain “officially planned at time of check” and require rechecking before implementation.
• The overall route’s nonexistent path was repaired. A specialist link audit found 0 missing/ambiguous wikilinks and 0 bad heading anchors for this phase’s core pages.
• Deep-reviewed pages fill content_origin/content_status; rerunning only local code uses execution_verified: 2026-07-19 without mechanically refreshing external source_checked.

Verification actually performed

Check	Result	Evidence boundary
11 vertical project test suites	1,191/1,191 passed	397 in each of normal, python -O, and warnings-as-errors: Document Parsing 23; Knowledge Base 29; Chunking 29; Embedding 30; Vector Database 28; Semantic Search 32; Reranking 29; RAG 68; Evaluation 36; LLMOps 60; Monitoring 33.
8 Python syntax checks	Passed	Built-in compile() after UTF-8 read; no bytecode.
Specialist red-team review	Passed	Rechecked citation/fact recall/trace-stage/public response/release fingerprint/time window/floating ref/duplicate ID/bulk audit/duplicate JSON key, and fallback misbinding/self-reference/blocked-unpromoted/cycle bypass. No residual high/medium finding.
CLI evidence recomputation	Passed	RAG normal PASS/fault BLOCK; protected Reranking envelope; LLMOps v6 promote/healthy audit 0 and cross-bind/no-window 2; Monitoring stable 0 and incident 1.
Document/local-link review	Passed	Consistent test counts, public/audit boundary, date fields, and terminology; missing/ambiguous wikilinks 0.
.website unit tests	33/33 passed	Publication policy, source gates, route metadata, path redaction, resource allowlist, repository scan.
Full .website build	Passed	880 source Markdown, 723 full pages, 157 stubs, 204 assets, 883 staged Markdown, 2,400 HTML pages.
Supplemental checks	Passed	git diff —check clean; tracked pycache/.pyc under projects 0; three outer local/submodule states untouched.

Key sources

This record’s source_checked summarizes Phase 4 review on 2026-07-19. Course-level external-source dates remain those stated in each page.

• Lewis et al., Retrieval-Augmented Generation, RAGAS, ARES, and BEIR: RAG baselines and heterogeneous-retrieval/evaluation limits.
• OWASP LLM08:2025 Vector and Embedding Weaknesses, OWASP LLM01:2025 Prompt Injection, MCP Client Concepts, and NIST AI 600-1.
• Google SRE, Implementing SLOs and Alerting on SLOs: SLI population, window, error-budget, and alert boundaries.

Follow-up queue

1. Create an end-to-end Layer B ingestion project connecting source → canonical document → chunk/span → index → RAG citation, and verify version migration, deletion propagation, and no revival of old snapshots.
2. Add strict MCP Resource/subscription validation, real AuthN/AuthZ, cancellation, and revocation propagation; Roots remain a coordination hint only.
3. Add a Tool Calling result envelope for untrusted-result schema/provenance/sensitive fields/status query/OUTCOME_UNKNOWN reconciliation.
4. Implement permission/deletion deep water: hierarchy, explicit deny, ABAC/provenance, index/cache invalidation, retention policy, and physical-purge evidence.
5. Bind complete SLO contract for Monitoring Layer B: population, exclusion rules, compliance window, data-source/query version, change approval, and missing-data policy.
6. Move from fixture digests to external artifact loading, signatures, attestations, approval principals, and immutable storage.
7. Continue route-metadata migration and add genuine RAG/evaluation/safety/LLMOps/monitoring hard prerequisites rather than inferring them from display order.
8. Add controlled tests with real vector databases, rerankers/LLMs, prompt injection/data poisoning, latency/cost, and Provider failures.
9. Replace teaching flags/deterministic IDs with real operator auth, random opaque trace IDs, redacted logs, and controlled audit store.
10. Manually inspect Mermaid, Callouts, long tables, and cross-course navigation in Obsidian/public site.

See the preceding Phase 3 record. Source/license policy continues to follow the Content Quality and Source Labeling Standard.