Data Annotation
About this knowledge base
Data annotation turns ambiguous business judgment into a repeatable, reviewable measurement protocol. It supports supervised training, RAG relevance assessment, Agent-trajectory review, safety red-teaming, and offline evaluation. The real deliverable is a “task definition + guideline + controlled input + annotation events + quality evidence + release manifest,” not a column of labels that merely appears complete.
Content and evidence boundary
This course is an original engineering synthesis based on the sources listed at the end. Tool interfaces, platform features, and regulations change; source links are for verifying concepts and tool behavior, not substitutes for a particular organization’s privacy, labor, contract, or legal review.
Where this course fits in the overall route
This course belongs to the Mathematics and Data Foundations stage. Study it after basic data cleaning; it supplies reliable supervision signals for Machine Learning, Reranking, RAG, Evaluation Systems, Data Synthesis, and production feedback. If the task touches high-stakes decisions, personal data, or harmful content, complete data-governance and safety boundaries before beginning a pilot annotation.
Master five objects first
| Object | Question it answers | What it must not stand in for |
|---|---|---|
sample_id | Which logical sample is being judged? | The exact input content or version |
source_revision | Which controlled input snapshot did the annotator actually see? | Source authorization, privacy safety, or factual correctness |
annotation_id | Who submitted one initial annotation under which protocol and when? | Ground truth or adjudication |
final_label | What label resulted from the established adjudication process? | Any projection used for training/evaluation |
release_id | Which frozen set of samples, contracts, quality gates, and splits will downstream users consume? | Permanent representativeness of future online distribution |
Keep guideline_version, label_set_version, and task_config_version with every annotation event as well. They respectively identify the rules, available labels, and UI/schema; a label with the same name need not measure the same object across versions.
Learning objectives
- Define annotation units, the target population, label ontology, exclusion conditions, and visible evidence.
- Write guidelines, paired examples, and escalation rules that unfamiliar annotators can execute.
- Control quality with pilots, overlapping annotation, blinded review, adjudication, and bridge samples.
- Interpret percent agreement and Cohen’s kappa correctly without conflating agreement, correctness, and representativeness.
- Treat active learning, weak supervision, and synthetic samples as candidate sources rather than unverified “true labels.”
- Manage the boundaries of sensitive data, licenses, worker exposure, versioning, withdrawal, and production feedback.
Prerequisites
Understand JSONL/CSV and sampling concepts, and be able to run a Python 3 standard-library script. It is recommended to first complete the sampling and uncertainty sections of Data Cleaning, JSON, and Probability and Statistics. Before starting, confirm least-privilege access, de-identification approach, source licensing, retention period, and escalation paths for harmful content.
Recommended order
- Task definition and annotation units — turn “is it good?” into an observable, adjudicable measurement task.
- Data governance, privacy, licensing, and worker safety — complete the minimum governance gate before handling real data or assigning it to real workers; return to other governance details after workflow practice.
- Annotation guidelines and edge cases — make ambiguity explicit with rules, paired examples, abstention, and escalation.
- Annotation workflow and data formats — bring samples, input snapshots, people, tools, and versions into one pipeline.
- Quality control, review, and adjudication — use independent judgments, bridge samples, and additive adjudication to handle disputes.
- Inter-annotator agreement metrics — calculate and diagnose agreement, reporting denominators, distributions, and uncertainty.
- Active learning, weak supervision, and human-in-the-loop work — improve efficiency without contaminating evaluation, amplifying bias, or anchoring human judgment.
- Versioning, release, and production feedback — turn annotation output into a traceable, withdrawable release that does not feed back into frozen sets.
- Project: Agent answer-quality annotation — run a strict two-annotator audit, explain conflicts, and write the next adjudication/release plan.
Hands-on project
audit_annotations.py runs strict JSON, input-snapshot, contract-version, and two-annotator-pair checks on sample_annotations.jsonl, calculating percent agreement, Cohen’s kappa, confusion counts, and conflicting samples. test_audit_annotations.py covers 12 normal and failure behaviors. The project needs no third-party package, real data, or API key.
Mastery checklist
- Can have two people who did not design the task independently perform a pilot, then turn ambiguity into versioned rules.
- Can distinguish “labels agree,” “labels are correct,” “the target population is represented,” and “it is safe to release.”
- Can choose quality evidence suitable for multiclass, sequence-labeling, or generative-rating measurement objects.
- Can trace every label to its input snapshot, guideline, label space, UI, annotator role, and time.
- Can explain how model pre-labeling, weak supervision, and synthetic candidates can anchor judgment or contaminate statistics.
- Can run normal/
-Omodes and all 12 tests, and explain why kappa is undefined when a constant label has perfect agreement.
Relationship to other knowledge bases
- Data Cleaning handles corrupt, duplicate, sensitive fields and controlled release manifests before annotation.
- Machine Learning consumes labels for training/evaluation but cannot exceed the bounds of their definition, split, and version.
- Evaluation Framework and Benchmark Design project adjudicated labels into frozen cases and rubrics;
cannot_judgeandexcludemust not be silently collapsed into positive/negative classes. - RAG and Reranking often need graded query-document relevance and evidence spans rather than simple binary labels.
- Data Synthesis can supplement candidates and edge cases; synthetic origin, generator version, and final human label must be stored in layers and cannot substitute for validation on a real distribution.
- AI Governance, Privacy Computing, and AI Safety constrain data sources, access, retention, affected people, and high-risk escalation.
Primary references
Sources were checked on 2026-07-22. The following are representative sources for concepts and tool behavior; a concrete project must lock its tool version, task protocol, organization policies, and applicable requirements.
- Label Studio: Set up and configure projects, import tasks, and export annotations
- Google PAIR: People + AI Guidebook
- Cohen, J. (1960). A coefficient of agreement for nominal scales
- Artstein, R., & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics
- Lewis, D. D., & Gale, W. A. (1994). A Sequential Algorithm for Training Text Classifiers
- Ratner et al. (2017). Snorkel: Rapid Training Data Creation with Weak Supervision
- Gebru et al. (2021). Datasheets for Datasets; Bender & Friedman (2018). Data Statements for NLP
- NIST AI RMF Core, NIST Privacy Framework, and W3C PROV-O
- ILO: Digital work and occupational safety and health