Quality Validation and Reproducible Pipelines
Objective
Turn cleaning into a layered, repeatable, regression-tested release process. Every acceptance, repair, quarantine, and overwrite action must be explainable from the input, rules, and report.
Four layers of a cleaning process
- Raw layer: retain received content, provenance, and checksums read-only.
- Normalized layer: standardize types, time, text, and enumerations without unexplained guesses.
- Valid layer: apply contract and cross-field rules and quarantine problematic records.
- Release layer: create a versioned snapshot for training, RAG, or monitoring.
Do not overwrite raw files in place. Each run records at least input identity, code/rule version, start time, row-count changes, and counts for every rejection reason.
Each consumable release batch also needs an immutable manifest: release_id, input snapshot or hash, data and schema version, rule/code version, group and split-mapping version, quality-gate result, owner, and generation time. The manifest proves what actually ran for this release. It cannot independently prove source authorization, privacy safety, or business correctness; each requires its own review evidence.
Write a release file completely to a temporary file in the destination directory before replacing the target, so a failed run does not leave a partial CSV. Cross-file transactions are difficult for multiple outputs, so consumers must also use the same batch ID or manifest and read only after the entire batch finishes.
Quality gates
| Gate | Example | Failure action |
|---|---|---|
| Schema | Required columns exist and types parse | Block release |
| Row level | Latency is nonnegative and status is enumerated | Quarantine row |
| Set level | Primary key is unique and missing rate is below threshold | Block or alert |
| Distribution level | Category, length, and latency show no abnormal drift | Human review |
| Relationship level | A chunk refers to a real document | Block ingestion |
Passing does not mean data is perfect; it means it meets the current explicit contract. Quality metrics must include their denominator, time window, and data version.
Repeatability and idempotency
The same input and configuration should produce the same output. A cleaning script must not depend on current time, directory order, or implicit global state. When a timestamp is needed, keep it as run metadata and do not let it alter data content.
Refusing to overwrite existing output by default prevents mistakes. If overwrite is allowed, require an explicit option and first confirm that input, output, and report resolve to different absolute paths. Idempotency does not mean source data may be overwritten at any time.
Test at least normal rows, each boundary, missing values, invalid types, duplicate primary keys, Unicode text, and an empty file. When production reveals a new failure, reduce it to an anonymized minimal regression case.
Machine-learning boundary
Deterministic normalization, such as time parsing and fixed-enumeration mapping, can run before the split. First fix entity, document, or session-level group_id and the frozen evaluation boundary so same-source content has not already crossed sets before a later “repair.” Any step that learns statistics from data—mean imputation, vocabulary, scaling, near-duplicate threshold calibration, or text-statistic features—must fit only on the training portion and be saved with the model Pipeline. Evaluation data may accept the same frozen deterministic parsing rules, but must not decide thresholds, vocabulary, or cleaning strategy in reverse.
Exercise
Design five blocking gates and three alerting gates for document ingestion. For each, specify the calculation, threshold, failure action, and owner.
Mastery check
- I can draw the raw, normalized, valid, and release layers and explain what each permits.
- I write a temporary file before replacing a target and understand that two files are not one transaction.
- I can prove byte-for-byte repeatability for the same input and rules, and show that failure cannot overwrite source data.
- I reduce a new production failure to a de-identified minimal regression example.
- I freeze group/split mapping in the release manifest and distinguish parsing that can run over all data from learned steps that can fit only on the training split.
Next: Project: Agent run-log cleaning.
References
Sources were checked on 2026-07-22.