AI Agent Engineer

AI Agent Engineer

Course-Wide Self-Check and Mastery

5 min read

Course-Wide Self-Check and Mastery

How to use this page

Close the course notes first and answer with your own examples. Each answer must contain three parts: definition or judgment, a concrete example, and an engineering consequence. Repeating terms alone does not pass. Then use the links in each section to check your answer and record concepts that remain confused.

Suggested scoring:

  • 0: cannot answer, or makes a key judgment error.
  • 1: the definition is roughly correct, but no example or engineering consequence is given.
  • 2: definition, example, and boundary are complete, including when the idea does not apply.

Proceed to the following main route only with a total score of at least 32 and no zero on a question marked Key. The score is for learning diagnosis, not a capability certification.

I. Concept map

  1. What is the difference between an AI system and a model? Why cannot responsibility be assigned to “the model itself”?
  2. What are the usual relationships among ML, DL, generative AI, LLMs, and Agents? Give one AI system that does not use an LLM.
  3. How do rule automation, one LLM call, a fixed workflow, and an Agent differ?

Check: AI Map and Core Terms.

II. Data, training, and generalization

  1. What are algorithms, models, parameters, training, and inference respectively?
  2. Among pre-training, fine-tuning, prompting, RAG, and tool calling, which usually change model parameters?
  3. How can a training objective, task metric, and system outcome become misaligned? Give an example with valid JSON but a business error.
  4. Key: Why is a test set no longer independent after repeated prompt tuning against it? What should be done?
  5. Give one example each of target leakage, temporal leakage, duplicate leakage, and evaluation contamination.
  6. Why is a random row split not always reasonable? Give a case that groups by user, document, or time.

Check: Data, Models, Training, and Inference and Generalization, Data Splits, and Leakage.

III. Evaluation and LLMs

  1. What can a baseline, frozen offline set, human review, shadow mode, and online monitoring each find?
  2. Key: Why can user likes, human acceptance, or scores from another LLM not directly become ground truth?
  3. How does an LLM generate output step by step from input text? What do tokens, parameters, context, and decoding each do?
  4. Why can lower temperature or more context not guarantee factual correctness?
  5. How do internal model token representations relate to, and differ from, text embeddings used for retrieval?

Check: Evaluation Evidence and Feedback Loops and How LLMs Generate Answers.

IV. Agent and failure boundaries

  1. Which system parts does an Agent add beyond one LLM call? Why are state, memory, and context not the same concept?
  2. Key: Why can a tool call not execute directly after its parameters pass JSON Schema?
  3. Why cannot prompt injection be solved only by “writing one more sentence saying do not be attacked”?
  4. What risks do idempotency, stop conditions, read/write separation, an identity-and-authorization chain, and human approval each control?
  5. When an answer is wrong, how do you distinguish failures in data, retrieval, context, generation, tools, and third-party dependencies?

Check: How Agents Complete Tasks and Capability Boundaries and Failure Modes.

V. Solution, launch, and responsibility

  1. Key: Given “let AI automatically choose the best candidates,” which questions would you ask first, and how would you narrow the scope?
  2. Why should the autonomy ladder begin with read-only information or drafts rather than direct automatic execution?
  3. What kinds of releases can a value gate, quality gate, and risk gate each block?
  4. How do shadow mode, degradation, rollback, and retirement differ?
  5. Why should a third-party model or tool service be re-evaluated after an update that causes no explicit error?
  6. Key: Under which conditions is human review an effective control?
  7. Why cannot trustworthy AI be compressed into one overall score? How can privacy, transparency, explainability, and fairness trade off?
  8. Why do high-impact contexts require professional review, access to evidence, and appeal paths?

Check: When to Use AI and Select a System Shape, From Prototype to Launch and Exit, and Responsible Use and Risk Controls.

VI. Integrated transfer questions

  1. For “internal policy question answering,” draw input, permissions, retrieval, model, citation validation, people, and logs, and mark trust boundaries.
  2. For that system, write a rules or search baseline, six test slices, one quality gate, and one retirement condition.
  3. For the same requirement, when should you use a fixed workflow, and when should you consider an Agent?
  4. If model performance improves but user completion rate falls, in what order would you investigate?
  5. If one high-consequence risk cannot be measured reliably, how should a team record and decide on it rather than assuming it is low risk?

These questions have no single technical answer, but every answer must have a clear goal, evidence, ownership, and safe exit. If an answer contains only “optimize the prompt,” “use a stronger model,” or “let people check it,” return to the relevant lesson and redo its exercise.

Mastery check

  • Can explain AI, ML, DL, LLMs, workflows, and Agents with personal examples.
  • Can distinguish data, parameters, training, inference, context, and feedback.
  • Can design development/test boundaries and critical slices with no obvious leakage.
  • Can attribute a failure to a system layer instead of vaguely saying “the model is not strong enough.”
  • Can compare rules, ML, LLMs, workflows, and Agents and select the minimum viable shape.
  • Can define a baseline, offline threshold, pilot, monitoring, rollback, and retirement.
  • Can explain when human oversight is effective and who is accountable.
  • Can design deterministic validation for output contracts, evidence, permissions, and external actions.
  • Completed Integrated Project: Meeting Action-Item Assistant and met the project rubric.

Next route

References

Accessed 2026-07-22.

Backlinks