AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

Evaluation Framework Learning Path

6 min read

Evaluation Framework

Course overview

Evaluation is not “running a model and getting a score.” It turns a product objective into repeatable tasks, data, grading, and decision evidence. Starting with an evaluation claim and its basic units, this course builds datasets; deterministic, human, model, and trace grading; layered RAG and Agent evaluation; statistical uncertainty; multi-dimensional release gates; and audit-and-governance workflows.

Fact boundary

Vendor evaluation APIs, grader types, and product interfaces change. This course extracts official methodology only. Sources were checked on 2026-07-22; the code project is entirely offline, uses no vendor API, and does not use marketing leaderboard figures. OpenAI announced the deprecation of its Evals platform on 2026-06-03. As of this page’s verification date, its stated plan is for existing evals to become read-only on 2026-10-31 and for the dashboard and API to close on 2026-11-30; those future dates must be rechecked when they arrive. This course does not teach that legacy entry point. MLflow links point to the documentation labeled latest when checked and do not represent a pinned package version; its classic-ML and GenAI evaluation APIs have different objects, metrics, and migration boundaries and must not be mixed.

Where this course fits

This course belongs to the “Production, Evaluation, and Governance” stage. Understand Probability and Statistics, RAG, and Agent Core first, then turn quality objectives into release gates and regression evidence. For long-lived designs of public comparison tasks, continue to Benchmark Design.

Learning objectives

  • State the claim, decision, object, and success criteria that an evaluation must support.
  • Build traceable datasets containing typical, boundary, failure, and adversarial scenarios.
  • Combine deterministic assertions, rules, human review, model grading, and trace grading, while stating each method’s boundary.
  • Interpret confusion matrices, precision/recall/F1, means, percentiles, and confidence intervals, distinguishing task-population estimates from trial variation.
  • Evaluate retrieval, generation, tool selection, parameters, traces, and final environment state separately.
  • Correctly distinguish offline evaluation from online experiments, reporting sample size, variation, and intervals.
  • Turn production failures into regression cases and control changes with versioned gates for safety, fairness, cost, latency, and quality.
  • Produce auditable reports that record data, graders, harnesses, approvals, exceptions, and evidence boundaries.

Prerequisites

  • Intuition for means, proportions, sampling, variance, and confidence intervals.
  • Ability to read JSON and run Python 3 standard-library scripts in PowerShell 7.
  • Familiarity with RAG’s retrieval-to-generation structure and an Agent’s tool-calling loop.
  • No specific evaluation platform is required; first learn transferable evaluation contracts.
  1. Evaluation Objectives and Basic Units — work backward from the decision an evaluation must support.
  2. Cases, Datasets, and Stratification — construct data that represents real use while retaining difficult tails.
  3. Deterministic Assertions, Metrics, and Scoring Rules — let machines verify objectively decidable conditions first.
  4. Human Review and Model-Based Evaluation — define judge calibration, bias, and the human-reserved boundary.
  5. Layered RAG, Generation, and Agent Evaluation — locate the layer at which a system failed.
  6. Offline, Online, Statistics, and Regression — move from a one-off score to reliable decisions.
  7. Evaluation Reporting, Audit, and Governance — turn multi-dimensional evidence into prioritized, reviewable release decisions.
  8. Offline-to-Online Evidence Handoff and the Regression Loop — connect frozen evaluation, release gates, telemetry, and human triage in a one-way, auditable evidence chain.
  9. Project: Offline Layered Evaluation Pipeline — run strict contracts, deterministic graders, slice gates, paired bootstrap, and a complete evidence summary.

Hands-on entry points

  • Main project: Offline Layered Evaluation Pipeline. Using only the Python 3 standard library, it validates strict dataset/rubric/prediction JSON contracts, split leakage, confusion matrices, slice gates, fixed-seed bootstrap, complete SHA-256 evidence summaries, display-only short fingerprints, and CLI exit codes. Observation fields in its fixtures are only offline teaching inputs; they do not constitute candidate self-attestation or real environment results.
  • RAG layered practice: Offline Citable Question Answering produces a versioned evaluation artifact with retrieval/context/citation fact recall, status accuracy, critical slices, and non-disclosure gates. It does not mistake a self-reported Boolean such as evidence_present=true for complete RAG evidence.
  • Reporting practice: use the generated Agent evaluation dashboard to inspect multi-trial proportion intervals, tail latency, routing confusion, and the cost-success tradeoff. Keep charts, fixtures, generator scripts, exact-dimension tests, and textual alternatives together.
  • Practical task: derive input, expected environment state, critical assertions, and failure severity from one real failure log.
  • Review exercise: have two reviewers blind-score the same output set, analyze disagreement first, and only then decide whether to introduce a model grader.

Mastery checklist

  • I can distinguish a task, trial, grader, assertion, trace, outcome, harness, and suite.
  • I can explain what claim a total score supports and what it cannot support.
  • I can document dataset source, stratification, version, privacy treatment, and freezing policy.
  • I can prefer deterministic checks and write anchored rubrics for subjective dimensions.
  • I can calibrate graders against a human gold standard and report agreement, disagreement, position/length/adversarial sensitivity, and adjudication.
  • I can diagnose retrieval, answer, tool parameter, and final-state failures separately rather than looking only at terminal text.
  • I can calculate and interpret a confusion matrix, precision, recall, F1, p95, and a confidence interval correctly.
  • I can make safety and critical-slice BLOCK outcomes take precedence over an overall mean and report fairness, cost, and latency limits.
  • I can report paired differences and uncertainty, add failure cases to regression candidates after human triage, and retain full audit evidence.

Relationships to other courses

  • Benchmark Design fixes long-term comparison protocols, task distributions, and reporting rules; this course first solves the claims, graders, data, and release gates for a particular product.
  • Data Synthesis can supplement rare cases, but synthetic samples must be labeled separately and their value validated against real data.
  • Runtime Monitoring continuously observes production telemetry, SLOs, and incidents. This course only turns candidates that have been human-triaged, de-identified, deduplicated, and confirmed reproducible into repeatable cases; it never writes monitoring curves or alerts directly into a frozen test set.
  • LLMOps integrates evaluation gates into versioning, release, and rollback workflows.

Primary references

All sources below are official materials, standards, or original papers and were retrieved or checked on 2026-07-21:

Backlinks