Harness Engineering: designing a legible, testable environment for coding agents

Original source

Ryan Lopopolo, OpenAI, February 11, 2026: Harness engineering: leveraging Codex in an agent-first world.

This is an original learning guide, not a reproduction of the article. Copyright and the complete work remain with the original source.

What you should be able to do after this guide

By the end, you should be able to:

1. Separate “can the model write this code?” from “does the environment let the Agent observe, act, and verify?”
2. Describe a coding harness through inputs, visible state, action interfaces, feedback, invariants, and stopping conditions.
3. Explain why isolated worktrees, a browser, logs, metrics, and traces form the Agent’s perception system rather than optional accessories.
4. Organize AGENTS.md, repository documentation, and code as a progressively disclosed knowledge map instead of endlessly enlarging one master prompt.
5. Turn architecture boundaries and engineering taste into executable checks whose failures also explain a remediation path.
6. Design a complete loop for a real repair—from reproduction and editing through acceptance, review, and post-merge cleanup.
7. Decide what level of autonomy a Harness can safely support and what risks still require human authorization or acceptance.

Start with the real problem: model capability is not completion capability

A model may know how to repair a null dereference and still fail the repository task. It may not know the startup command, may be unable to see the real page, may share a database with another task, may receive interleaved logs, may face architecture rules that exist only in a maintainer’s head, or may have no way to tell whether green tests correspond to correct product behavior.

Those failures should not automatically be labeled “the model is not smart enough.” The model performs reasoning and generation; the environment determines what evidence it can obtain, what actions it can execute, and how it can know that work is actually finished. Practical capability emerges from both:

$$\text{effective engineering capability}\approx \text{model capability}\times \text{environment legibility and feedback quality}$$

This is an engineering mnemonic, not a measured equation from the article. It highlights a simple failure mode: if either factor approaches zero, the whole system performs poorly. Harness Engineering focuses on the second factor—designing repositories, tools, knowledge, constraints, and feedback so an Agent can use them independently.

The mechanism in one sentence

When Agents perform much of the implementation, the engineer’s scarce contribution shifts from typing every line to making goals, knowledge, runtime state, tools, architecture boundaries, acceptance evidence, and continuous cleanup understandable to the Agent and mechanically verifiable as a closed loop.

flowchart TD
    I["Humans define goals, boundaries, and risk authorization"] --> P["Repository knowledge map and task plan"]
    P --> A["Agent observes and edits in an isolated worktree"]
    T["Terminal, browser, logs, metrics, and traces"] --> A
    A --> V["Tests, lint, architecture checks, and real-app acceptance"]
    V -->|"Failure evidence + remediation direction"| A
    V -->|"Stopping contract satisfied"| M["Commit, review, and merge"]
    M --> O["Production observation and bad-pattern collection"]
    O --> C["Garbage collection: docs, rules, structure, and dependencies"]
    C --> P

What is a Harness? Write down six contracts first

A Harness is not one prompt or CLI. It is the complete action environment an Agent relies on from receiving a task to proving completion. A useful audit begins with six contracts:

Contract	Question it must answer	Examples
Input	What must the Agent achieve, and what may it not change?	Issue, acceptance criteria, risk limits, relevant links
Visible state	Which real states can the Agent observe?	Files, git diff, page, database, logs, metrics, traces
Action interface	What may the Agent execute safely?	Search, edit, test, start app, use browser, create PR
Feedback	How does each action reveal progress or regression?	Exit code, failed assertion, screenshot, error log, structural check
Invariants	Which boundaries must no implementation bypass?	Dependency direction, authorization, data isolation, API compatibility
Stopping conditions	What evidence proves completion, and when must work stop?	Target and regression acceptance; escalate after budget exhaustion

If a task says only “fix this Issue,” while the other five contracts remain implicit tribal knowledge, the Agent must guess. A good Harness externalizes that knowledge without prescribing each line of implementation.

Layer 1: make the application itself legible to the Agent

Give every task an isolated runtime scene

The article describes an environment in which tasks use separate worktrees and their application instances, logs, and observability data correspond to those worktrees. A dependency or database change made by Agent A should not present Agent B with an unexplained symptom.

Isolation should answer at least:

• Which commit and worktree belong to this task?
• Which port, database, and temporary storage does it use?
• How do logs, metrics, and traces correlate to this task?
• How are processes started, reset, and stopped?
• Which state must be preserved at the end, and which state should be discarded?

“Several Agents run in parallel” does not prove isolation. Shared ports, caches, test databases, or log streams contaminate feedback and can make one Agent repair a symptom caused by another task.

A browser is an operable product surface, not merely a screenshot tool

For a Web product, unit tests alone cannot prove that an interaction works. The Agents in the article can start the application and use browser capabilities, including interfaces such as the Chrome DevTools Protocol, to inspect pages, click, type, and observe runtime state.

Feedback can then expand beyond “the test command is green” to:

1. Does the page actually load?
2. Do UI state and URL change after a user action?
3. Does the console emit an error?
4. Do the network request, backend log, and trace belong to the same operation?
5. Does the state remain correct after refresh or restart?

The goal is not to give the Agent more buttons. It is to connect a user-visible symptom to code, requests, and backend state.

Logs, metrics, and traces are the Agent’s senses

A human investigating “saving is slow” may inspect the browser, server logs, database metrics, and a distributed trace together. An Agent given only a one-line error summary lacks the same diagnostic surface.

Observability must be queryable, correlated, and scoped. Task ID, worktree, request ID, and time window should connect one run across systems; output should be noise-controlled and must not reveal credentials. More logging does not automatically improve legibility. Stable queries and structured fields are more useful than one enormous mixed terminal stream.

Layer 2: make the repository the system of record for knowledge

AGENTS.md should be a map, not an encyclopedia

The article describes keeping a short entry document on the order of roughly one hundred lines and treating it as a table of contents into architecture, product, plans, quality, and operations material. The durable idea is not an exact line limit. It is a stable, compact entry with details disclosed on demand.

One practical hierarchy looks like this:

AGENTS.md
├─ Stable global boundaries and common commands
├─ Architecture map → docs/architecture/
├─ Business domains → docs/product/
├─ Plans and decisions → docs/plans/, docs/decisions/
├─ Testing and release → docs/quality/, docs/runbooks/
└─ Local instructions → module/AGENTS.md

The Agent reads the entry to decide where to look, then loads only the relevant layer. This avoids both no context and thousands of rules competing for limited attention at once.

Why repository knowledge should be the system of record

If critical knowledge lives only in chats, meetings, or an external page, an Agent cannot reliably discover it during work or know which version is current. Keeping plans, decisions, and operational instructions in the repository lets code and its knowledge dependencies be versioned, searched, reviewed, and checked together.

Writing text into the repository does not automatically make it true. Documents need an owner, scope, or verification path. If code changes while instructions remain stale, the Agent may treat obsolete prose as authority. The Harness should therefore check links, runnable commands, and important code–documentation contracts.

Layer 3: optimize for Agent legibility, not only human readability

Agent legibility means a model can construct an accurate local system model from the nearby files. It often comes from:

• directories aligned with business domains instead of every behavior living in generic utils;
• stable dependency directions and explicit cross-layer entry points;
• names that express domain concepts and traceable state transitions;
• discoverable configuration, generation steps, and side effects;
• short non-interactive commands for common tasks;
• error output explaining what happened, where it happened, and what to inspect next.

This does not require rewriting all code “for the Agent,” nor does it mean more comments are always better. The goal is to reduce implicit conventions known only to long-tenured maintainers. Stable abstractions and clear domain boundaries help both humans and Agents.

Maps and constraints have different jobs

Documentation explains how the system is organized, why a boundary exists, and where to learn more. Executable checks answer whether that boundary still holds now. Prose alone drifts; checks alone leave the Agent without design intent or a valid repair path.

Layer 4: turn architecture and engineering taste into executable feedback

Enforce invariants without dictating every implementation detail

The article emphasizes custom lint, structural tests, and CI to enforce architecture rules. “The domain layer must not depend on the UI layer” is a strong candidate for a machine check. “This function must use a three-level factory pattern” is usually over-specification.

A useful constraint is:

1. Stable: it represents a durable boundary, not a one-off preference.
2. Decidable: a tool can inspect dependencies, ASTs, configuration, or runtime results and return a clear result.
3. Remediable: a failure names the reason, affected location, and compliant route.

BAD: architecture check failed
 
BETTER: packages/domain/order.py imports packages/ui/table.py.
Domain code may depend only on domain and platform contracts.
Move the display conversion into packages/ui/adapters/; see
docs/architecture/dependencies.md.

The second message is both a quality gate and a feedback signal for the Agent’s next iteration. It protects the boundary while leaving implementation choices to the task context.

How engineering “taste” becomes mechanical over time

When a team observes a repeated bad pattern, it can first record a golden principle, then decide whether automation is warranted. For example:

• service code must not bypass the authorization entry point;
• tests must not depend on the public Internet;
• every new state needs a clear owner and cleanup path;
• user-visible failures must retain a diagnosable cause.

Not every principle can or should become lint immediately. Calibrate its meaning through reviews and examples. Once the pattern is stable and false positives are controlled, turn it into a check. Otherwise “automated taste” merely blocks legitimate design.

Layer 5: make the feedback loop self-closing

An autonomous coding loop is not just “generate code, then run all tests.” It should:

1. Read the task contract and repository map, then form a minimal plan.
2. Reproduce the current failure in an isolated environment and preserve baseline evidence.
3. Trace the execution path and make the smallest relevant change.
4. Run the most targeted fast check and revise from specific failure output.
5. Start the actual application and observe the user path, logs, and traces.
6. Run regression, architecture, type, formatting, and security-boundary checks.
7. Review the final diff for unintended files, debug code, and out-of-scope changes.
8. Produce an evidence-backed commit or PR; machines and people continue acceptance according to risk.
9. Observe the merged behavior and turn repeated defects into documentation, tests, constraints, or cleanup work.

def run_task(task, harness, max_attempts=4):
    workspace = harness.create_isolated_workspace(task)
    contract = harness.load_task_and_repository_map(task)
    baseline = harness.reproduce(workspace, contract)
 
    for _ in range(max_attempts):
        change = harness.agent.edit(workspace, contract, baseline)
        feedback = harness.run_targeted_checks(workspace, change)
 
        if feedback.failed:
            baseline = feedback
            continue
 
        acceptance = harness.run_full_acceptance(workspace, contract)
        if acceptance.proves_done:
            return harness.prepare_review(workspace, acceptance)
 
        baseline = acceptance
 
    return harness.escalate_with_evidence(workspace, baseline)

The critical field is proves_done. Stopping conditions come from the task contract and external state, not from the Agent saying the work “should now be fixed.” Budget exhaustion also needs an explicit exit instead of infinite retries.

Walk through one task: from “the Save button does nothing” to a mergeable repair

The following end-to-end example is for teaching; it is not an internal Issue from the article.

1. A human defines the goal and boundaries

The Issue states: after editing a profile, clicking Save gives no feedback; success should display a notice and refresh the name; authorization behavior must not change; both success and failure paths need coverage. The owner allows frontend, backend, and test edits, but any database migration requires human approval.

2. The Harness creates a reproducible scene

• Create an isolated worktree from the designated commit.
• Allocate an independent port and temporary database.
• Start the frontend, API, and observability services.
• Associate the browser session, request IDs, logs, and traces with this task.

The Agent reproduces the problem in the browser, saves a failure screenshot or video, and sees a 403 response. The trace shows that authorization middleware reads an old field name. This is much closer to the root cause than the statement “the button is broken.”

3. The Agent follows the knowledge map to the contract

AGENTS.md points to the account domain and authorization-boundary documentation. The architecture map says the UI must not construct authorization fields itself. The Agent finds a recent field rename and compatibility layer, confirming that the repair belongs at the API adapter boundary rather than bypassing authorization.

4. Edits receive feedback in layers

The Agent first adds an API regression test that fails, then fixes the compatibility mapping. After the target test passes, it saves again through the browser: a success notice appears, the name refreshes, the console remains clean, and the trace returns 200. It then runs authorization-failure cases, account-module tests, type checks, and architecture checks.

If the Agent tries to hard-code an administrator token in the UI, a security or architecture check should fail clearly and point to the authorization adapter. The Harness does not need to predict the correct code, but it must block this invalid shortcut.

5. Produce a reviewable delivery

The PR contains the failure baseline, root cause, minimal diff, target and regression results, successful page evidence, and a statement that no database change occurred. A person can focus on authorization risk and product semantics instead of rediscovering the whole execution scene.

6. Perform garbage collection after merge

If the same field rename has caused similar failures in three modules, the team should not simply merge a third patch. It should converge on a shared adapter, remove the old field, add a structural check, and update migration documentation. Future Agents should not continue copying the bad precedent.

How higher throughput changes merge philosophy

The article describes a shift toward smaller, faster changes once Agent throughput rose and follow-up repairs became cheaper. This can shorten branch lifetime and reduce the conflicts associated with large PRs.

It is not a universal rule that high throughput removes review. Merge policy still depends on reversibility, test and monitoring quality, and whether a change affects migration, security, billing, or irreversible external actions. A low-risk UI copy change and an authorization-system migration should not receive the same autonomy.

What “Agent-generated” really means

In the article, Agent-generated material includes not only product code, but tests, CI, documentation, observability, and internal tooling. Humans still:

• define product goals and acceptable trade-offs;
• design the environment in which Agents work;
• encode architecture and security boundaries;
• resolve ambiguity, authorize risks, and own outcomes;
• notice when the system begins accumulating bad patterns.

“No application code was manually typed line by line” therefore does not mean “no human engineering,” and it does not erase responsibility. Human work shifts toward system design, result evaluation, and long-term coherence.

Autonomy is a ladder, not a switch

Level	What the Agent may do	Minimum Harness required	Human responsibility
0: advise	Read and propose a plan	Searchable code and documentation	Decide whether to implement
1: edit	Modify a constrained branch and produce a diff	Isolated workspace, scoped permissions, diff audit	Review every change
2: verify	Start, test, and iterate independently	Reproducible runtime, structured feedback, budget limits	Accept high-risk judgments
3: submit	Create an evidence-backed PR	Stable CI, ownership, and rollback path	Risk-tiered sampling or approval
4: merge/deploy	Complete pre-approved low-risk categories	Enforced policy, monitoring, fast rollback, audit	Define authorization and handle exceptions

Before raising autonomy, prove that the environment can observe success and failure—not merely that the model occasionally writes correct code. Permissions, external side effects, and cost budgets must also narrow explicitly at each level.

Why high output creates entropy—and how continuous garbage collection responds

Agents learn from patterns already present in the repository. If two temporary bypasses exist, the next change may introduce a third. If documentation is stale, an Agent can spread the obsolete approach faster. Higher output can also accelerate duplicate utilities, abandoned abstractions, dependency drift, and naming forks.

The article’s response is not an occasional heroic rewrite but continuous small-scale garbage collection:

1. Collect repeated bad patterns from review, failures, and code search.
2. Decide whether each is an isolated defect or a system gap.
3. Encode stable conclusions as golden principles, documentation, shared abstractions, or mechanical checks.
4. Remove obsolete paths, combine duplicates, and repair docs in small batches.
5. Observe false positives and real effects; revise or remove a rule when necessary.

This is maintenance of the Harness itself. Otherwise the Agent simply works faster inside an increasingly illegible environment.

How to read the article’s evidence

The article reports an internal case that began from an empty repository. After roughly five months, it describes approximately one million lines of code and about 1,500 Pull Requests, with average throughput around 3.5 PRs per engineer per day. It says the work took roughly one tenth of the traditional estimate, that the product served hundreds of internal users, and that application code was not manually written line by line.

Those figures show the throughput observed by this team under its models, tools, product, and counting method, and they make the engineering practices worth studying. They are not a controlled experiment: there is no randomized control group, and lines of code or PR counts do not directly measure product value, correctness, or long-term maintenance cost.

Evidence boundaries

• “Roughly one tenth of the time” is the team’s estimate in the article, not a fixed multiplier transferable across organizations.
• “Zero manually written application code” describes a production method, not an absence of human prompting, review, architecture judgment, or tool building.
• Internal users and a five-month observation window show real usage, but cannot establish multi-year coherence or maintenance cost.
• This is a 2026 case from one team. Models, Codex capabilities, and organizational processes continue to change; reuse the mechanisms rather than copying the numbers.

Common failure modes

Failure mode	Surface symptom	Root cause	Harness improvement
Giant instruction file	Many rules exist, yet the Agent misses them	No navigation hierarchy; attention diluted	Short entry + layered docs + local rules
Fake isolation	Parallel tasks randomly affect one another	Shared ports, database, cache, or logs	Bind runtime resources to each worktree
Green tests only	The page is still broken or state does not persist	Acceptance omits the real product path	Browser, restart, and end-to-end state checks
Architecture prose only	Boundary violations keep appearing	Rule is not executable	Structural tests, lint, CI, remediation message
Over-constraint	Small changes repeatedly hit irrelevant rules	Personal preference encoded as invariant	Automate stable boundaries only; monitor false positives
Lots of unusable logs	Agent searches text but cannot find the request	No correlation fields or scope	Task/request IDs, structured queries, noise reduction
Premature autonomy	Agent merges a high-risk mistake	Weak stopping, rollback, and permission contracts	Risk-tiered authorization with human gates
Entropy growth	Duplicate tools, stale docs, and temporary bypasses spread	New-feature throughput dominates maintenance	Recurring garbage collection and golden principles

Historical contribution and today’s engineering interpretation

The article’s contribution is not a new model algorithm. It makes “agent-first software engineering” concrete as repository and environment practices: application legibility, layered in-repository knowledge, executable architecture, closed-loop feedback, bounded autonomy, and continuous entropy cleanup.

The mechanism you can adopt today is to find states the Agent cannot independently observe or verify, then add the smallest tools and contracts that expose them. Turn repeated human corrections into repository knowledge or machine constraints. Increase autonomy only when real terminal states can be proven and failures can be rolled back.

The article does not establish that every team should ban manually written code, that more PRs or lines necessarily create more value, that a short AGENTS.md replaces architecture design, or that tests and lint eliminate human ownership.

Relationship to neighboring readings

SWE-agent shows that an immediate Agent–Computer Interface can materially change repository-task performance. Building Effective Agents separates predictable workflows from autonomous Agents. Effective Context Engineering explains how to manage limited attention. Harness Engineering extends these interface, control-flow, and context ideas across the repository and software lifecycle.

Check that you can apply the mechanism

1. A model can propose the right patch but cannot see the page or logs. Is that a model-capability failure or a Harness failure? Which contracts are missing?
2. Why is putting all knowledge into a 2,000-line AGENTS.md not equivalent to “more complete context”? What organization is better?
3. Which rule is a better candidate for a mechanical check: “the domain layer cannot depend on UI” or “use a factory pattern”? Why?
4. The Agent’s target tests and lint pass, but data disappears after refreshing the page. Can it declare completion? What is missing from the stopping contract?
5. When may an Agent move from “create a PR” to “automatically merge low-risk changes”? Name the minimum conditions.
6. Why does a high-throughput team need a fixed garbage-collection rhythm even more?

Reference answers and judgment criteria

1. The model may already have generation capability; the failure is mainly in the Harness’s visible-state, action, and feedback contracts. Add browser operation, correlated logs/traces, a reproducible runtime, and real acceptance.
2. A giant entry consumes attention, mixes scopes, and goes stale. Keep stable boundaries and a map in the entry, then link to architecture-, business-, task-, and directory-specific material disclosed on demand.
3. Dependency direction is a stable invariant decidable from an import graph, so it is a better check. A factory pattern is usually an implementation choice unless it carries a proven durable boundary.
4. No. The checks omit persistent user state. Add the real browser path, state after refresh or restart, and supporting backend evidence to the stopping contract.
5. Restrict it to pre-approved low-risk classes and require stable CI, real acceptance, scoped permissions, monitoring, audit, and fast rollback. Security, migration, and irreversible actions still need a human gate.
6. Agents rapidly reproduce existing repository patterns, so output also accelerates duplication, drift, and stale knowledge. Small recurring cleanup converts repeated corrections into shared rules and preserves legibility.

Previous: Effective Context Engineering · Back to the landmark reading index