Runtime Monitoring Learning Path

Runtime Monitoring

Course overview

Runtime monitoring tells a team whether a service meets user expectations and helps it locate, contain, and learn from anomalies in latency, errors, quality, safety, or cost. Monitoring usually checks predefined signals and conditions; observability emphasizes answering questions that were not enumerated in advance from a system’s outputs. Both depend on meaningful instrumentation, semantics, and operational processes.

This course covers logs, metrics, traces, instrumentation and context propagation, SLIs/SLOs, quality/safety/cost signals, alerts, dashboards, drift investigation, and incident response. It does not replace the model lifecycle in MLOps, the LLM release unit and evaluation gates in LLMOps, or the evaluators defined by Evaluation Framework. Instead, it monitors the coverage, trends, and action boundaries of those online signals.

Where this course fits

Runtime monitoring sits between “deployed” and “operated reliably.” Learn APIs, JSON, logging, model/LLM call paths, and basic evaluation first. The resulting operational evidence feeds later work in evaluation, benchmark design, AI safety, and AI governance.

Learning objectives

After completing this course, you will be able to:

• distinguish the questions suited to logs, metrics, and traces, and correlate them with IDs, resource attributes, and time;
• design automatic and manual instrumentation, W3C Trace Context propagation, and Collector pipelines, including monitoring of the telemetry path itself;
• derive an SLI, SLO, statistical window, and error budget from user expectations;
• organize service and resource symptoms with RED, USE, and the four golden signals, and correctly interpret histograms, percentiles, sampling, and high cardinality;
• combine availability and tail latency with AI task quality, safety, cost, and label coverage in one monitoring surface;
• design actionable alerts with suppression, runbooks, clear owners, and recovery conditions;
• build overview → drill-down → evidence dashboards instead of decorative walls of charts;
• correctly interpret drift, label delay, sampling, and missing signals;
• run a full incident loop: classification, containment, evidence preservation, recovery, and postmortem learning.

Prerequisites

• Read and write JSON with Python 3, and understand averages, percentiles, ratios, and time windows.
• Understand HTTP status codes, request IDs, timeouts, retries, and batch jobs.
• Have a basic grasp of model versions, RAG, Tool Calling, and LLM token/cost accounting.
• You do not need Prometheus, OpenTelemetry, or a cloud-monitoring platform installed first.

Recommended sequence

1. Logs, Metrics, and Traces — understand the tradeoffs and correlation of the three signal types.
2. Instrumentation, Collectors, and Correlation — keep telemetry correlated and governable across HTTP, asynchronous jobs, and tools.
3. SLIs, SLOs, and Error Budgets — select signals and objectives from user expectations.
4. Quality, Safety, and Cost Signals — make AI failures visible beyond HTTP 200.
5. Alert Design and On-Call Operations — send actionable symptoms to the right people only.
6. Dashboards and Diagnostic Views — create a drill-down path from user impact to one trace.
7. Drift, Feedback, and Label Latency — treat distribution change as the start of an investigation, not an automatic root cause.
8. Incident Response and Postmortems — protect users, preserve evidence, and verify recovery under pressure.
9. Offline Monitoring Audit Project and Self-Check — validate structured events, traces, RED/USE, SLOs, error budgets, and multi-window alerts.

Hands-on entry point

The main project is Offline Monitoring Audit Project and Self-Check. It uses only Python’s standard library. It strictly validates a small JSON event, trace, and release-evidence contract; turns raw invalid UTF-8, Unicode surrogates that cannot be stably encoded as UTF-8 scalar sequences, and numbers that cannot be represented as finite float values into controlled errors; calculates RED/USE, SLIs, p95, bad-event ratios, and error-budget burn rates from an explicit window_end; and never calls a burn rate remaining budget. It separately checks the age of business events and Collector exports. Release evidence explicitly declares the instructional format of the candidate-gate summary it accepts. Anomalous windows produce only a no-raw-content regression candidate for human triage; they never rewrite a frozen evaluation set automatically. Forty-five tests cover these boundaries. All IDs and thresholds are offline teaching data: no network, third-party package, or key is needed.

Mastery checklist

• I can drill from a slow request metric to its trace and structured logs.
• I can explain W3C Trace Context, a Collector pipeline, semantic-convention stability, sampling, and cardinality boundaries.
• I can write an SLI with an event definition, good-event condition, window, and data source.
• I can distinguish technical success, task quality, safety outcome, cost, and observability completeness.
• I can give an alert user impact, severity, owner, runbook, suppression, and recovery conditions.
• I can explain the evidence limits introduced by sampling, high cardinality, label delay, and proxy metrics.
• I can turn an incident postmortem into a new alert, regression sample, runbook, and exercise.
• I can run the offline project and its 45 tests on Windows 11/PowerShell 7, and explain why every threshold is a local teaching policy.

Relationships to other courses

• MLOps uses runtime signals to assess model drift, release, and retraining; this course owns signal semantics, SLOs, alerts, and incident processes.
• LLMOps defines LLM release units, trace metadata, and Canary decisions; this course supplies the general observability and response method.
• Evaluation Framework defines task quality and evaluators. Runtime monitoring tracks online coverage, trends, and failure samples, then outputs only human-triage candidates rather than writing a frozen evaluation set automatically. The full cross-stage handoff fields, summary, and human-triage boundary are specified in Offline-to-Online Evidence Handoff and Regression Loop.
• AI Safety defines threats and controls; runtime monitoring operationalizes only observable safety symptoms, coverage, and response.

Primary references

All official or primary sources below were checked on 2026-07-22. Standard stability, tool syntax, and support status can change, so implementation must pin a version and recheck it. The OpenTelemetry core semantic-conventions page was at 1.43.0 when checked, while GenAI conventions have moved to a separate repository. A production contract must pin the repository revision, schema URL, or equivalent contract version in use, determine stability from the actual signal and component, and run compatibility tests. Do not mistake the core-page version for a current GenAI field version.

• OpenTelemetry Observability primer — observability, reliability, and signal fundamentals.
• OpenTelemetry Signals — official entry point for logs, metrics, and traces.
• OpenTelemetry Collector Architecture — Receiver, Processor, and Exporter pipelines.
• OpenTelemetry Versioning and stability — stability must be assessed by component.
• OpenTelemetry Generative AI semantic conventions — migration note on the core site.
• OpenTelemetry GenAI semantic conventions repository — current separate specification repository; pin a specific revision in implementation.
• W3C Trace Context — standard for propagating trace context across services.
• Google SRE: Service Level Objectives — choosing SLIs/SLOs and error budgets from user behavior.
• Google SRE: Monitoring Distributed Systems — latency, traffic, errors, saturation, and related fundamentals.
• Google SRE Workbook: Alerting on SLOs — error-budget burn rates and multi-window alerts.
• Prometheus Alerting practices — symptom orientation, actionability, and monitoring the monitoring system.
• NIST SP 800-61 Rev. 3 — current NIST resource for incident response and risk management.