AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

Benchmark Design Learning Path

5 min read

Benchmark Design

Course overview

A Benchmark is a versioned, repeatable evaluation contract used for comparison. It is not “a set of questions plus an average.” It must also define the claim it supports, target population and task space, data sources and splits, run protocol, baselines, metrics, statistical reporting, submission rules, and maintenance ownership.

This course is for engineers designing a team Benchmark for the first time. The intended deliverable is not a polished leaderboard, but an evidence system that can answer: “Who performs better, under which conditions, on which tasks, and at what cost?”

Fact and version boundary

Public leaderboards, models, and platforms change continuously. This course does not record rankings or performance figures that expire quickly; methods and sources were checked on 2026-07-21. MLPerf rules use the current content of the official repository at that check date as an example; actual submissions must verify the then-current rules. Stanford CRFM’s official repository states that HELM entered maintenance mode on 2026-06-01. This course continues to cite the original HELM paper for methodology, but does not describe the framework as a default tool actively receiving new functionality.

Where this course fits

This course belongs to the “Production, Evaluation, and Governance” stage. Complete Evaluation Framework first to learn cases, graders, and regression, then learn to freeze them into a fair, reproducible, maintainable Benchmark for long-term comparison. Data Synthesis can provide synthetic stress samples, but must not replace evidence about the real distribution.

Learning objectives

  • Write a Benchmark claim, scope, and explicit non-goals from the decision it must support.
  • Check representativeness with task spaces, stratification matrices, and gap tables rather than question count.
  • Design group-level splits, frozen test sets, and a contamination register; distinguish training contamination, runtime discoverability, development adaptation, and maintenance-chain leakage.
  • Freeze baselines, prompts, decoding, tools, budgets, and environment so comparison conditions are equivalent.
  • For Agent tasks, freeze initial/final state, side effects, reset, timeout, retry, and multi-trial rules.
  • Report primary metrics, risk thresholds, stratified results, resource cost, and statistical uncertainty together.
  • Recognize leaderboard gaming, repeated tuning, and Benchmark saturation, and maintain evidence boundaries through report cards, version changes, and retirement.

Prerequisites

  • Ability to read JSON and run Python 3 standard-library scripts in PowerShell 7.
  • Basic intuition for accuracy, means, proportions, sampling, and confidence intervals.
  • Understanding of task, trial, grader, outcome, and suite from Evaluation Framework.
  • Neither model training nor a real API is required.
  1. From Claim to a Benchmark Contract — decide what decision a score must support.
  2. Task Space, Representativeness, and Stratification — expand real use into a coverage matrix.
  3. Data Splits, Leakage, and Contamination — protect independent evidence and document contamination risk.
  4. Baselines, Metrics, and Comparable Runs — freeze system boundaries and the run protocol.
  5. Reproducibility, Statistics, and Results Reporting — report uncertainty, per-item results, and reproducibility materials.
  6. Leaderboard Mechanics, Anti-Gaming, and Maintenance — limit speculation with submission policy, private testing, and version governance.
  7. Agent Environments, State, and Repeated Runs — freeze tools, initial/final state, side effects, reset, and multi-trial protocol.
  8. Project: Build a Maintainable Benchmark — run strict JSON, protocol comparability, critical-task gates, and paired intervals.

Hands-on entry points

  • Main project: Build a Maintainable Benchmark. The fully offline project demonstrates PASS, a critical-task BLOCK, INCOMPARABLE protocol, and data-contract error separately.
  • Design exercise: choose one real Agent task and write a one-page Benchmark card with its claim, non-goals, task space, risk thresholds, and update conditions.
  • Audit exercise: take a public leaderboard and list at least five questions that rank alone cannot answer.

Mastery checklist

  • I can distinguish an evaluation suite, Benchmark, competition, and leaderboard.
  • I can state the target population, task distribution, critical strata, and coverage gaps.
  • I can create group-level splits based on a generating unit such as user, document, template, or time.
  • I can explain why public tests become contaminated, distinguish not-detected from unknown, and state why contamination detection provides only risk evidence.
  • I can freeze equivalent protocols for all systems and report quality, risk, cost, and latency together.
  • I can define environment, tools, initial/final state, side effects, reset, budget, and multiple trials for an Agent case.
  • I can produce a result package with a version fingerprint, sample count, per-item results, intervals, and limitations.
  • I can design rules for submission rate, private tests, audit, report cards, retirement, and migration to a new version.

Relationships to other courses

  • Evaluation Framework provides cases, graders, human review, and regression methods; this course owns the fair comparison contract across versions or systems.
  • Probability and Statistics supplies foundations for intervals, sampling, and hypothesis testing.
  • Data Synthesis can fill combinatorial conditions and rare risks, but must retain a synthetic label and be calibrated against real holdouts.
  • AI Governance determines who approves scope, metrics, public disclosure, and high-risk conclusions.
  • Runtime Monitoring provides distribution drift and failure samples that trigger a Benchmark update rather than indefinite static reuse.

Keep the boundary clear: Evaluation Framework mainly supports development, release, or regression decisions for one concrete product. Benchmark Design fixes a longer-lived comparison protocol across systems. Runtime Monitoring observes actual production traffic, SLOs, and incidents. They can feed data back to one another, but a production monitoring curve must not be presented as a frozen-Benchmark result.

Primary references

All sources below are official materials, standards, or original papers, retrieved or checked on 2026-07-21:

Backlinks