AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

LLMOps Learning Path

7 min read

LLMOps

Course overview

LLMOps is the engineering practice that makes a Large Language Model (LLM) application evaluable, releasable, traceable, observable, and reversible. It operates not only a model file but a composite application: application code; API/SDK contract; prompt; context assembly; retrieval snapshot; model and parameters; tools; routing; data and safety policy; evaluators; and price-accounting convention.

The smallest control loop is:

Fixed release manifest → offline evaluation gate → human approval → Canary
        ↑                                             ↓
Regression suite and policy updates ← online observation/feedback ← degrade, roll back, or continue

This course focuses on controlling one LLM-application change with evidence. For prompt writing see Prompt Engineering, for retrieval principles see RAG, and for model training, registry, and deployment pipelines see MLOps.

Where this course fits

LLMOps sits between “can call an LLM” and “can operate an LLM application over time.” Learn LLM API integration, prompting, context, RAG, Tool Calling, basic MLOps, and Evaluation Framework first; at minimum you must define an objective, build a dataset, and set a release gate. Continue with runtime monitoring, linking AI safety and AI governance into the same evidence chain.

Learning objectives

  • Define a complete LLM-application release unit and trace every key version from a production output.
  • Use a gateway to centralize authentication, rate limits, timeout, retry, routing, and cost attribution.
  • Make evidence-based caching and routing decisions across quality, latency, cost, and data boundaries.
  • Trace model, retrieval, and tool calls while limiting sensitive-content exposure.
  • Combine regression suites, deterministic checks, model grading, human review, and statistical uncertainty into a release gate.
  • Recognize provider failures, API changes, and behavioral drift, and switch only to prevalidated fallback paths.
  • Design Canary, human approval, rollback, incident evidence collection, and review.

Prerequisites

  • Read and write JSON in Python 3, and understand HTTP, API-key placeholders, timeout, and error handling.
  • Know prompts, tokens, context windows, Embeddings, RAG, and Tool Calling.
  • Understand Git commits, immutable versions, offline evaluation, and progressive release.
  • No cloud platform or Kubernetes experience is required.
  1. Release Units and Version Lineage — answer “what exactly is running in production?”
  2. Gateways, Quotas, and Resilience — converge provider calls at a controlled entry.
  3. Caching, Latency, and Cost — distinguish optimizations that improve the end-to-end task.
  4. Traces and Privacy Boundaries — trace every decision in a request without retaining unnecessary content.
  5. Offline Evaluation Gates and Regression Suites — decide whether a change can reach production using task evidence.
  6. Canary, Rollback, and Change Management — limit blast radius and preserve a way back.
  7. Continuous Evaluation, Provider Drift, and Governance — turn online feedback into auditable decisions rather than automatic rewrites.
  8. Incident Response and Continuous Improvement — improve tests, alerts, and release policy from failure timelines.
  9. Offline Release Gate Project and Self-Check — run candidate gate, online observation gate, and audit tests without a real API.

Hands-on entry point

The main project is Offline Release Gate Project and Self-Check. It uses only the Python standard library. It rejects duplicate JSON fields, raw invalid UTF-8, surrogate values that cannot encode stably as UTF-8 scalar sequences, and values not representable as finite float. It binds evaluation artifact to subject release; validates baseline/candidate comparison before binding Canary with full candidate_gate_evidence_sha256, display-only short fingerprint, promoted_at, fixed decision_as_of, and concurrent control. artifact_digest_format and the observation bundle’s evidence_digest_format jointly declare this repository’s versioned local teaching byte profile; the bundle schema/profile also enters every online decision’s full digest. Do not mistake this for a signature or universal canonical JSON. Online quality and safety results derive from verifiable integer numerators and denominators, where zero denominator means unknown. Automated fallback must match the gate-bound fallback release identity and full digest, have sufficient current candidate evidence, and reject self-reference, known cycles, and known failing fallback candidates. The 74 tests cover bad binding, full-digest tampering, format mismatch, schema/profile binding, pre-release/future windows, fallback-evidence bypass, ratio-bound errors, numerator/denominator consistency, insufficient evidence preventing automatic fallback in fixtures/CLI, label coverage, and INCOMPARABLE. Observation can become continue, investigate, pause, fallback, rollback, or human_review.

You can reuse the data → script → PNG → alternative-text loop in the Agent Evaluation Dashboard, but a release gate remains based on raw trials, slices, and gate evidence; never infer or hand-edit data from a chart.

On Windows 11 and PowerShell 7 this project runs directly. If you need an isolated interpreter, create it outside the vault:

$llmopsVenv = Join-Path $env:TEMP 'llmops-learning-venv' # Keep the isolated environment outside the vault.
python -m venv $llmopsVenv # Create the standard-library Python environment used by this project.
& "$llmopsVenv\Scripts\Activate.ps1" # Pin subsequent python/pip to the isolated interpreter.
python -m pip --version # Verify pip/interpreter only; install no package.
python ".\docs-EN\llmops\examples\release_gate.py" audit --release-id release-safe --observation-id obs-healthy # Audit a fixed candidate and healthy observation.

The project has no third-party dependency. Do not create .venv in the knowledge base or write real keys.

Mastery checklist

  • I can name the code, API, prompt, context, retrieval, model, tools, policy, evaluation, routing, and price-accounting versions behind one online answer.
  • I can distinguish rate limiting, concurrency control, timeout, retry, circuit breaking, fallback, and capacity reservation.
  • I can explain different risks of prefix caching, exact-answer caching, and semantic caching.
  • I can use a trace to locate whether model, retrieval, tool, or retry caused a latency/cost anomaly.
  • I can block a regression in important task, safety slice, or cost with a regression suite and release gate.
  • I can write Canary expansion, pause, fallback, and rollback conditions while handling delayed labels.
  • I can explain why provider drift must not trigger an unvalidated automatic model switch.
  • I can run ordinary, -O, warnings-as-errors, and combined tests and explain each blocking reason.

Relationships to other courses

  • MLOps supplies general traceability, evaluation-gate, release, and rollback ideas; LLMOps extends the release unit to a composite LLM application.
  • Runtime Monitoring owns Logs, Metrics, Traces, SLI/SLO, and alerting platform. LLMOps defines which LLM versions and quality evidence enter observation. A full evidence digest may be a Trace/Log or controlled audit field, never a Metric label.
  • Evaluation Framework designs samples, metrics, and scorers. LLMOps versions and connects them to release and continuous evaluation. Online anomalies first enter controlled human triage; an alert or dashboard may not rewrite a frozen evaluation suite automatically.
  • AI Safety and AI Governance define threats, approval, audit, and accountability. LLMOps makes them concrete in gates, runbooks, and evidence chains.

Version and source notes

This route, offline project evidence, and the external material below were checked on 2026-07-22. Provider APIs, model aliases, prices, product UI, and retention capabilities can change; production implementation must recheck current documents and contracts. OpenAI appears as a provider example only and is not a required LLMOps platform.

Current OpenAI documentation marks older Evals/Graders products as in a deprecation-migration period. This course therefore uses general principles — task-specific evaluation, continuous evaluation, human calibration, and fixed scorer version — rather than treating an old product interface as a stable dependency. Current MLflow documentation distinguishes classic-ML mlflow.models.evaluate()/EvaluationMetric from GenAI mlflow.genai.evaluate()/Scorer, which are not interchangeable. The OpenTelemetry core semantic-conventions page is currently 1.43.0, while GenAI conventions moved to a separate repository; integration must pin the actual revision/schema URL and test compatibility by signal/component stability instead of applying the core-page version mechanically.

Primary references

Backlinks