Modern LLM Capabilities and Model Selection

Course positioning

Model selection is neither copying a leaderboard nor buying the “largest model” by default. It is a constrained engineering decision: first express the task, risks, and deployment boundary as a capability contract and hard gates; then run repeated trials for every candidate with the same harness, data, and configuration; finally, analyze the Pareto trade-offs among quality, cost, latency, and risk.

This course does not maintain model rankings. Model catalogs, prices, context limits, regional availability, and preview status are all volatile facts; it teaches a selection method that transfers across them.

Boundary for dynamic facts

Sources were checked on 2026-07-22. On the day of a decision, vendor model IDs, capabilities, quotas, prices, data policies, and retirement dates must be reverified from official catalogs, model cards, contracts, and actual probes, with the evidence URI, time, and configuration retained. An advertised context window is an integration limit, not proof of effective task-level context capability; public benchmarks are evidence for candidate discovery, not a production conclusion.

latest, preview aliases, and pinned versions do not have the same change semantics. Even if a model ID identifies fixed weights, routing, safety layers, or service infrastructure can still change observable behavior. Record the requested identifier; the model/version actually returned in the response when the API provides it; the endpoint/region; and the adapter version. Treat vendor lifecycle changes or runtime changes as triggers for new probes and regression tests.

Learning objectives

• Translate a business task into contracts for input/output, reasoning, context, structured output, tools, and modalities.
• Distinguish an API’s claimed support from reliable behavior on this task and target configuration.
• Select matching evaluation objects and metrics for embeddings, generative models, and multimodal models.
• Build hard gates from data residency, retention, training use, deployable environments, licenses, budget, and latency.
• Run repeated trials on the same case with globally unique trial_id values, reporting failure distributions, cost, and finite-sample statistics instead of treating different cases as repetitions.
• Eliminate ineligible candidates before computing interpretable scores, a Pareto frontier, and weight sensitivity within the eligible set.
• Produce a reviewable model-decision record and define regression and rollback conditions for version upgrades.

Prerequisites

• Read and write JSON, and run Python-standard-library scripts and unittest in PowerShell 7.
• Understand HTTP APIs, timeouts, JSON Schema, and basic statistics; prior integration with a particular vendor is not required.
• This course is an entry point in LLM application foundations and can be studied alongside Prompt Engineering, Context Engineering, and LLM API Integration. When you begin probing real candidates, add provider adapters, usage accounting, and retries.
• If you select an embedding model, continue to Embeddings as needed. If you select an Agent model, use Tool Calling to verify the complete action contract.

Recommended order

Order	Lesson	Key artifact
1	Capability contracts and the selection object	One-page task-to-capability contract
2	Reasoning, effective context, and structured output	Three executable acceptance cases
3	Tool support and Agent runtime compatibility	Tool/runtime compatibility matrix
4	Multimodal, embedding, and specialized models	Modality pipeline and embedding evaluation set
5	Hard gates, privacy, and deployment boundaries	Auditable gate table
6	Task-level multi-trial evaluation	Frozen case, trial, and metric protocol
7	Pareto trade-offs and sensitivity analysis	Frontier, weight perturbations, and decision record
8	Project: an auditable model-selection scorecard	Offline scorecard, failure cases, and test evidence

Plan for 12–16 hours: 60–90 minutes for each of the first seven lessons, plus 3–5 hours for the project and review.

Evidence flow from candidates to release

flowchart LR
    A[Task, risks, and success evidence] --> B[Candidate discovery]
    B --> C[Evidence completeness and hard gates]
    C -->|FAIL or BLOCKED| X[Stop, obtain evidence, or approve an exception]
    C -->|PASS| D[Repeated trials with a common harness]
    D --> E[Slices, costs, and failure distributions]
    E --> F[Pareto frontier and declared weight scenarios]
    F --> G[Auditable decision record]
    G --> H[Release gates, monitoring, and rollback]

The sequence in the diagram represents evidence dependencies, not a waterfall process: when the task, policy, candidate, or runtime changes, rerun the affected upstream steps. BLOCKED means evidence is missing and cannot be offset by a total score.

Learning artifacts

After finishing the course, retain five versioned artifacts:

1. task contract: task distribution, inputs and outputs, risks, and success evidence;
2. candidate evidence: model IDs, API capabilities, model cards, policies, and prices, together with parseable verified/blocked status, owner, expiry, and missing items;
3. eval protocol: frozen cases, trial count, grader, configuration, and stopping rules;
4. decision report: gate results, metric distributions, Pareto frontier, sensitivity, and approved exceptions;
5. release guard: selected version, regression thresholds, canary, monitoring, and rollback conditions.

Mastery criteria

• I can explain why “supports 1M context” does not prove reliability on a long-document task.
• I can express JSON Schema, tool, and multimodal support as runnable contract tests.
• I can select queries, corpus, relevance, and latency metrics for an embedding that match the retrieval task.
• I can run hard gates for privacy, deployment, capability, cost, and latency before comparing scores.
• I can distinguish case_id from a globally unique trial_id and verify that every candidate reaches the preregistered repetition count for every case.
• I can explain that nearest-rank p95 from a small sample describes only recorded samples and does not prove stable tail latency.
• I can distinguish the questions answered by a total score, a Pareto frontier, and sensitivity analysis.
• I can reject the argument “it ranks first on a public leaderboard, so deploy it directly.”
• I can run the project tests and explain which fixture candidates should be eliminated by gates and which are Pareto dominated.

Connections to other courses

• Evaluation Framework provides the full method for tasks, trials, graders, traces, and outcomes; this course narrows it to model-candidate decisions. After the scorecard passes, use Offline-to-Online Evidence Handoff and Regression Loop to connect the summary, release gate, operational evidence, and human triage.
• Benchmark Design explains public comparison protocols; this course prioritizes private task distributions and deployment constraints.
• LLMOps owns model-version release, canaries, monitoring, and rollback.
• Retrieval-Augmented Generation (RAG) and Agent Core provide end-to-end acceptance objects for generative, retrieval, and tool environments.
• Privacy Computing and AI Safety deepen data, identity, supply-chain, and misuse risks.

Primary sources

The following sources establish the method; this course does not copy volatile leaderboard values from them as conclusions:

• Stanford CRFM, HELM: a reproducible, transparent framework for foundation-model evaluation and HELM Long Context
• Liang et al., Holistic Evaluation of Language Models
• NIST, AI 600-1: Generative Artificial Intelligence Profile
• Mitchell et al., Model Cards for Model Reporting
• Muennighoff et al., MTEB: Massive Text Embedding Benchmark
• JSON Schema 2020-12 specification
• Examples of entry points for dynamic verification: OpenAI model catalog, Gemini API model catalog and version naming, Claude model catalog, and Claude model IDs and version semantics. These pages are for evidence collection on the integration date, not vendor recommendations.