Probability and Statistics

Course overview

Model outputs, sampled evaluations, production success rates, and retrieval metrics are all random to some degree. Probability gives you a language for describing uncertainty; statistics lets you infer properties of a population from limited data. This course begins with units of analysis, denominators, and descriptive statistics, then develops probability models, sampling, and inference before applying them to Agent A/B evaluation. The goal is not to memorize formulas. It is to avoid mistaking one run, one average, or one p-value for a certain conclusion.

Where this course fits

Probability and Statistics belongs to the Mathematical and data foundations stage. It supplies the language of uncertainty used throughout machine learning, evaluation systems, benchmark design, and runtime monitoring. You can begin after the engineering foundations; the same ideas recur whenever you compare models, retrievers, or Agent versions.

Learning objectives

• Update judgments with events, conditional probability, and Bayes’ rule.
• Describe data with means, medians, quantiles, standard deviations, and stratified tables while checking denominators, missingness, and duplicates.
• Recognize random variables, distributions, expectation, variance, and sampling variation.
• Express uncertainty in a metric with a confidence interval.
• Interpret hypothesis tests, p-values, multiple comparisons, and effect sizes correctly.
• Define a target population, unit of analysis, pairing structure, primary metric, and minimum practically important difference for Agent evaluation.
• Compare two Agent versions with a standard-library bootstrap and state the evidence boundary honestly.

Prerequisites

Basic arithmetic, square roots, Python lists, and functions are enough. The summation symbol $\sum$ means “add a collection of numbers.” Complete Python Fundamentals first if possible. This course does not require calculus or a third-party statistics package.

Recommended sequence

1. Descriptive statistics and data quality: first establish what a row represents, what the denominator is, and whether data are missing or duplicated.
2. Probability, conditional probability, and Bayes’ rule: begin with uncertain events and base rates.
3. Random variables and common distributions: map success, categories, counts, and latency to probability models.
4. Expectation, variance, and sampling: understand why average metrics vary and why correlated observations do not equal more information.
5. Estimation, confidence intervals, and hypothesis testing: draw appropriately limited conclusions from a sample.
6. Evaluation design, effect sizes, and statistical pitfalls: fix the population, unit, pairing, primary metric, and decision threshold before calculating.
7. Project: Agent evaluation uncertainty: complete a tested paired percentile-bootstrap comparison.

Hands-on entry points

• Start with an Agent-evaluation data health check in Descriptive statistics and data quality.
• Continue to Project: Agent evaluation uncertainty to implement a paired percentile bootstrap with the standard library, then use replicated samples and stratified reporting to recognize false certainty.

Mastery checklist

• Distinguish populations, samples, parameters, statistics, and random variables.
• Explain why $P(A\mid B)$ and $P(B\mid A)$ are not interchangeable.
• Report sample size and dispersion or an interval alongside a mean.
• Explain why “fail to reject the null hypothesis” does not prove two systems are the same.
• Choose a paired evaluation and report an interval for the difference with bootstrap.
• Identify whether independence, representativeness, and distributional assumptions hold.
• Specify the primary metric, minimum practically important difference, and stopping rule before looking at results.

Relationship to other courses

Course	Connection
Machine Learning	Probability describes prediction, loss, noise, and generalization uncertainty.
Evaluation Framework and Benchmark Design	Populations, sampling, pairing, intervals, effect sizes, and multiple comparisons set the boundary of a conclusion.
Embeddings and Semantic Search	Retrieval metrics are random variables; thresholds and distributions depend on queries and corpora.
Runtime Monitoring	Success rates, latency quantiles, and alerts require stable baselines and time-dependence analysis.
Data Visualization	Plots reveal distributions, strata, outliers, and temporal structure; a summary number is not enough.

Primary references

• NIST/SEMATECH e-Handbook of Statistical Methods
• NIST Exploratory Data Analysis
• NIST Probability Distributions
• NIST Hypothesis Tests and Confidence Intervals
• NIST Randomized Block Designs
• ASA Statement on Statistical Significance and P-Values
• Python statistics and random

Sources checked on 2026-07-14. This course uses NIST, ASA, and the Python 3.14.6 documentation as its external baseline; the local examples were verified with Python 3.11.9. A source list cannot replace review of the current sampling process and data structure: statistical-method validity depends on both.