AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

Document Parsing

5 min read

Document Parsing

Course overview

Document parsing converts the bytes, pages, and layout in a file into structured elements that can be retrieved, cited, and verified. For RAG, obtaining “one long string of text” is not enough: heading hierarchy, reading order, table cells, page numbers, coordinates, permissions, and parsing warnings all affect later chunking and answer evidence.

This course follows an important boundary: use deterministic detection and native parsing first, and use OCR or vision models only when needed; every probabilistic result must retain its provenance and quality state. The learning project does not pretend that the standard library alone can parse PDF or Office files. Instead, it demonstrates how to register, route, and reject them safely.

Where this course fits

This is an entry point in the RAG and knowledge bases stage. Complete Data Cleaning and JSON first, then study this course. Its parsing output is versioned by Knowledge Base Construction and composed into retrieval units by Chunking Strategies.

Learning objectives

After completing this course, you should be able to:

  • distinguish filename extensions, transport Content-Type, magic bytes, and inspection inside containers;
  • design a replayable pipeline of discovery, registration, detection, parsing, normalization, acceptance, and publication;
  • handle UTF-8, BOMs, line endings, and Unicode normalization with strict decoding;
  • preserve headings, lists, code, tables, reading order, and provenance coordinates;
  • distinguish the capability boundaries of digital PDFs, scanned PDFs, hidden text layers, OCR, and VLMs;
  • set gates for type, size, count, time, permissions, and isolation for untrusted files; and
  • determine usability with automated metrics and human gold sets instead of merely checking whether a process exited successfully.

Prerequisites

  • Know how to use Python strings, dictionaries, exceptions, pathlib, and unittest.
  • Understand media types and requests/responses in API.
  • Understand the basic structure of Markdown, JSON, and CSV.
  1. Format identification and the parsing pipeline: build a safe routing contract from bytes, filenames, and content signals.
  2. Encoding, text, and normalization: avoid mojibake, silent character loss, and unauditable “cleanup.”
  3. Structure, headings, lists, and tables: turn body text into elements with hierarchy, order, and coordinates.
  4. PDF and OCR boundaries: choose native extraction, OCR, layout models, or human review by page.
  5. Metadata, quality validation, and the project: run the offline inspector and interpret pass, review, and fail gates.

Use a rhythm of “read one lesson, complete its exercises, change one project rule, and run the tests.” Do not install a large parsing framework first. Understand the input, output, failure, and evidence contracts before deciding whether a framework result is trustworthy.

Hands-on entry point

Project files:

It scans a specified directory read-only and produces deterministic manifests for UTF text, Markdown, HTML, CSV, and JSON. It stops at external_adapter_required for PDF, Office files, archives, and images. Its 27 tests cover format disguises, strict decoding, duplicate JSON keys, CSV embedded newlines, parse revisions, source line numbers, content and element hashes, symbolic links, and resource budgets. If a file grows after the stat() precheck, bounded reading rejects it rather than reading the whole file into memory.

The focused project validates only a parser manifest. First study the Source-to-citation reference model for target invariants. Then, in the Cross-module source-adaptation project, call the current parser fresh and handle native line coordinates, namespaced element IDs, and a release crosswalk. External Provenance Artifact v2 continues by validating whether a parser record, raw/canonical representations, and a crosswalk can be rebuilt fresh across a strict JSON boundary.

Mastery criteria

  • I can explain why a .pdf suffix, an HTTP Content-Type, or one magic pattern cannot independently prove a file type.
  • I can design an element schema for headings, paragraphs, tables, and code that includes source_id, order, location, and hashes.
  • I will not disguise a decoding failure as success with errors="ignore".
  • I can decide whether a page needs native extraction, OCR, vision understanding, or human review.
  • I can explain why OCR confidence is not a probability that a fact is correct.
  • I can make identical inputs and configuration produce the same manifest, and produce a new version ID when the source file changes.
  • I can design automated gates, stratified samples, and human acceptance for critical fields.
  • I can state clearly which format capabilities the project has and has not verified.

Connections to other courses

  • Knowledge Base Construction: manages raw files, parsing versions, failure queues, and publication state.
  • Chunking Strategies: composes parsed elements and should not repeat expensive OCR.
  • OCR: covers image preprocessing, text recognition, and OCR evaluation in depth.
  • Multimodal AI: handles charts, formulas, visual question answering, and other tasks beyond text-only parsing.
  • RAG: retrieval and answer quality are bounded by parsing completeness and traceability.
  • AI Safety: untrusted documents can contain malicious containers, scripts, prompt injection, and sensitive information.

Primary references

Sources retrieved on 2026-07-22. Tool versions change; when installation options or supported formats matter, consult the current official documentation.

Backlinks