AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

Data Cleaning

4 min read

Data Cleaning

Course overview

Data cleaning is not “removing rows that look abnormal.” It transforms raw data into interpretable, verifiable, and traceable input under an explicit data contract. Dirty data in Agent, RAG, and evaluation systems appears as parsing failures, duplicated knowledge, incorrect timelines, inflated metrics, and irreproducible results.

Where this fits in the overall path

This course is in the Mathematical and data foundations stage. It builds on Python, JSON, and regular expressions, and directly supports machine learning, knowledge-base construction, data annotation, evaluation, and runtime monitoring.

Learning objectives

  • Write a schema and quality threshold for a table or JSON event.
  • Distinguish missing, invalid, duplicate, conflicting, and statistically unusual data.
  • Normalize text, time, categories, and numeric values while retaining raw data and an audit record.
  • Build repeatable cleaning processes and automatic quality checks.
  • Distinguish a schema, a cross-team data contract, and a consumer-facing release manifest; understand that they cannot replace one another.
  • Complete an Agent run-log cleaning project with no third-party packages.

Prerequisites

You should be able to use basic Python 3 syntax and understand CSV, JSON, fields, and data types. Complete Python Fundamentals, JSON, and Regular Expressions first if possible. Learn venv + pip first; the project in this course uses only the standard library and requires no additional package.

  1. Schema and quality thresholds: define what is valid before discussing repairs.
  2. Missing values and missingness mechanisms: make missing-data treatment follow business meaning and evaluation boundaries.
  3. Duplicates and entity identity: prevent duplicate counting, retrieval contamination, and data leakage.
  4. Outliers and range rules: distinguish invalid data, rare but real events, and distribution drift.
  5. Text, JSON, and time normalization: handle the semistructured data most common in Agent engineering.
  6. Quality validation and reproducible pipelines: replace visual spot checks with quality reports and gates.
  7. Project: Agent run-log cleaning: run the complete cleaning, quarantine, and reporting workflow.

Hands-on entry point

clean_agent_runs.py reads dirty_agent_runs.csv, validates the schema strictly, normalizes status and UTC timestamps, rejects duplicate IDs, validates latency, and writes clean data and a privacy-aware issue report separately. test_clean_agent_runs.py covers ten normal, boundary, and failure behaviors. The script refuses to overwrite its source; an existing output is replaced only with an explicit —overwrite option.

Mastery criteria

  • I can state a field name, type, nullability, allowed range, uniqueness rule, and unit.
  • I can explain why missing values must not always become zero and outliers must not always be deleted.
  • I can distinguish row-level duplicates, entity duplicates, and semantic near-duplicates.
  • I can record input count, output count, rejection reasons, and rule version in a cleaning report.
  • I can freeze entity or document-level grouping and an evaluation split before learning statistics, and explain how it prevents leakage.
  • I can ensure that repeated runs with the same input and rules produce the same result.
  • I can run normal and -O modes plus the ten tests, proving that the source file is unchanged, output is byte-for-byte reproducible, and caches are cleaned.

Connections to other knowledge bases

Primary references

Sources were checked on 2026-07-22. The pandas stable documentation is currently in the 3.0 series and will continue to change. This course’s project does not depend on pandas; its examples were verified with Python 3.11.9. When using pandas or a data-quality framework, pin the project version and rerun contract tests.

Backlinks