2026-07-22 Phase 23 Deep Review Record: Safety, Governance, and Production Evidence Chain

Phase conclusion

This phase unifies safety, governance, evaluation, release gates, and runtime monitoring into a reviewable production evidence chain. Risk/control requirements cannot be replaced by a model output, hash, Trace ID, or passing boolean. Offline evaluation, release approval, Canary window, and regression candidate each have their own version, complete digest, owner, and independent source/authorization boundary.

It covers AI Safety, AI Governance, Evaluation Systems, LLMOps, and Runtime Monitoring.

What this phase does not prove

Offline JSON, fixed fixtures, SHA-256, single-machine tests, and a static-site build cannot prove real model/tool execution, artifact provenance, Provider compliance, identity authorization, signed approval, production-telemetry completeness, user ground truth, causal business effect, or legal compliance. They prove only that the current teaching contracts, negative paths, handoff fields, and public-release boundary can be checked repeatedly.

Multi-Agent division of work and primary-Agent review

1. The Safety/Governance specialist checked AI Act timelines, NIST/MCP security boundaries, identity/least privilege/supply chain, red teaming/incident response, traceability, and production change page by page. The Governance project adds data/Provider owners and version fields.
2. The Evaluation specialist checked MLflow automatic evaluation, OpenTelemetry GenAI semantic conventions, RFC 8785 boundaries, and the offline evaluation evidence chain. The primary Agent made invalid UTF-8 and oversized numeric values fail as CLI contract errors.
3. The LLMOps/Monitoring specialist narrowed conservative responses to Provider drift, observation-window/business-event age, integer numerators/denominators, and regression-candidate boundaries.
4. Two independent read-only reviews checked Safety/Governance and the cross-course digest protocol. The latter identified and drove fixes for online-observation bundle schema/profile not entering the decision digest, escaped surrogate rejection being too late in evaluation, and format labels not being fixed by golden vectors.

The worktree already contained uncommitted work in this phase’s scope. This phase did not revert or reattribute it; it supplemented, narrowed, and verified issues discovered through contextual checks.

Key changes

Safety and governance: controls, approvals, and evidence no longer impersonate one another

• The AI Safety course connects identity, least privilege, supply chain, red teaming, monitoring, and incident response into an auditable control chain. A hash can find a mismatch only when actually compared in a controlled release gate; it cannot prove authorization or provenance. An MCP token likewise is not user intent, object authorization, or approval.
• The AI Governance course corrects boundaries between the EU AI Act’s in-force and future-applicability dates and says AI Omnibus must still be checked by its independent procedural status. NIST AI RMF 1.0 revision status, transparency/documentation, release approval, and change management are no longer presented as static interchangeable facts.
• The Governance project adds data/Provider owner, version, and approval evidence. Both Safety and Governance projects expand negative contract tests so field presence cannot be mistaken for an implemented control.

Evaluation, release, and monitoring: a versioned local profile, not fictitious universal canonical JSON

• The Offline Layered Evaluation project rejects raw invalid UTF-8 and isolated surrogate escapes at input time, and treats oversized values that cannot be represented as finite float as contract errors. Complete digests, format labels, and fixed golden vectors have regressions.
• Three projects explicitly share python-json-sorted-utf8-v1, a local Python-teaching byte profile only. It places positional arguments in an outer array, serializes JSON with sorted keys, compact separators, and allow_nan=False, then UTF-8 encodes and calculates SHA-256. It is not RFC 8785 JCS, cross-language canonical JSON, a signature, or proof of provenance.
• The LLMOps eval artifact uses artifact_digest_format; the online-observation bundle uses evidence_digest_format. The bundle’s schema_version and format field now both enter every online Decision.evidence_sha256, so an upgrade cannot leave unbound interpretive semantics. Tests cover unknown format, format-label drift, and schema/profile changes.
• Runtime Monitoring accepts candidate_gate_evidence_digest_format and separately records the runtime-summary format in Decision and human-triage handoff. Fixtures remain independent teaching samples and must not pretend to interoperate end to end with real LLMOps artifacts.
• The Offline-to-Online Evidence Handoff and Regression Loop lesson was added/rewritten with a field-mapping table and Mermaid evidence chain. It makes explicit that a monitoring finding can create only a candidate awaiting human triage; it cannot automatically write back to the frozen evaluation set.

Production feedback: stop automatic expansion when evidence is insufficient

• Provider drift always enters HUMAN_REVIEW when fallback binding is incomplete or the candidate lacks current-window evidence; an observation boolean cannot automatically trigger FALLBACK.
• The Monitoring project uses explicit window_end to distinguish business-event age from Collector-export age. A fresh Collector does not prove the business stream is current; stale events can escalate to PAGE.
• Online quality/safety ratios derive from verifiable integer numerators/denominators. A zero denominator is unknown, not zero or success. A non-OK window emits only a candidate without original text and awaits human triage.

Cross-course terminology and evidence boundaries

Term	Stable meaning in this phase	It must not be substituted for
evidence_sha256	A change-detection reference under a declared algorithm, version, and input bytes	Signature, provenance proof, approval, authorization, or real execution receipt
python-json-sorted-utf8-v1	A constrained Python-teaching byte profile shared by three offline projects	RFC 8785 JCS or any cross-language canonical JSON
artifact_digest_format	The declared digest-byte profile of an evaluation artifact	The artifact has been loaded, recomputed, or stored trustworthily
candidate_gate_evidence_digest_format	The monitoring-side declaration of release-gate digest format	Upstream release-gate fact or real release provenance
monitor_evidence_digest_format	The runtime-window digest format declaration in a human-triage candidate	Online result is trustworthy or approved for regression-set writeback
HUMAN_REVIEW	A freeze on automated expansion when evidence is insufficient, handed to an accountable reviewer	Automatic fallback, automatic approval, or unattended recovery

Sources, original material, and third-party boundary

All new Chinese-language explanations, Mermaid diagrams, field mappings, teaching code, and tests in this phase are original. The pages summarize facts and engineering boundaries from primary specifications/official materials only; they do not copy text lacking explicit redistribution permission. Principal checks:

• NIST AI RMF Core, NIST AI 600-1 Generative AI Profile, NIST SP 800-61 Rev. 3, and the European Commission AI Act framework.
• MCP Authorization, MCP Security Best Practices, and RFC 8785 JCS.
• OpenAI Evaluation best practices, MLflow automatic evaluation, OpenTelemetry Semantic Conventions, OpenTelemetry GenAI conventions, and Google SRE SLO guidance.

Verification evidence

Check	Result
AI Safety project	80 tests: normal, -O, warnings-as-errors, and -O + warnings-as-errors all passed.
AI Governance project	73 tests; all four modes passed.
Offline Evaluation project	43 tests; all four modes passed.
LLMOps release-gate project	74 tests; all four modes passed.
Offline Monitoring project	45 tests; all four modes passed.
Consolidated project regressions	315 tests per mode; 1,260 test executions across four modes all passed.
Static checks	In-memory compile() of 10 scoped Python files, related JSON/Markdown semantics, and scoped git diff —check passed.
Website tests	.website npm test: 48/48 passed; npm run build succeeded.
Publication validation	925 public pages and 231 assets; 0 broken local links, prohibited files, sensitive leaks, or KaTeX errors.

Follow-up queue

1. Connect offline eval/artifacts, release gates, Canaries, and runtime windows to one controlled artifact repository or API. Recompute digests in practice, verify provenance/signature/approval, and run clearly specified interoperability tests across language implementations.
2. Before connecting real models, tools, Collectors, and evaluation backends in an isolated environment, design redaction, cost caps, least privilege, sample permission, rollback, and human-escalation paths. Do not connect teaching fixtures directly to production.
3. In the next phase, deeply review the RAG core path, MCP/Tool Calling, Agent Skills, framework practice, and MLOps. Continue Obsidian-desktop reading checks of long tables, Mermaid diagrams, and callouts.

This record covers only Phase 23. It does not mean the long-running /goal is complete.