SWE-agent: why the Agent-Computer Interface changes performance

Original source

John Yang et al., 2024: SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. This guide uses arXiv v3.

This is an original learning guide, not a reproduction. Copyright and the complete paper remain with the original source.

What you should be able to do after this guide

• Explain in your own words why an Agent-Computer Interface (ACI) is more than a list of tools.
• Trace the complete loop from an issue through localization, inspection, reproduction, editing, testing, and patch submission.
• Explain the separate jobs of command design, feedback formatting, context management, and guardrails.
• Compare a custom ACI with a general-purpose shell: both can operate on a repository, so why can they still lead the model to different results?
• Design a controlled experiment that changes the ACI without changing model weights, then state precisely what the experiment can establish.

The idea in one sentence

SWE-agent’s central contribution is not “yet another agent that writes code.” It treats the language model as a new kind of computer user: holding the model fixed while changing the actions it can issue and the feedback it receives can materially change software-engineering performance.

Start with the problem: if the shell works, why build an ACI?

Suppose the model receives a real GitHub issue: “The configuration parser incorrectly accepts a negative timeout. Fix it and add a regression test.” With an ordinary shell, it can theoretically complete the task using ls, grep, cat, sed, Python, and test commands. The problem is that “expressible in principle” is not the same as “reliably usable by a model”:

• grep may emit thousands of lines and bury the three relevant ones;
• printing an entire large file rapidly consumes context;
• sed and redirection can edit files, but multiline changes are vulnerable to escaping mistakes and unintended replacements;
• when a successful command produces no output, the model may mistake silence for failure;
• an invalid edit without syntax checks and explicit feedback can corrupt every later step;
• if old tool output remains in the history, the model may keep reasoning from code that has already changed.

People use IDEs not because a shell is incapable of changing files, but because an IDE organizes search, navigation, editing, syntax feedback, and execution for a human user. The paper asks the corresponding design question for language models: models have their own strengths and limitations, so they may need a purpose-built interface too.

Three terms that are easy to confuse

Term	Meaning in this guide	Not the same as
LM agent	A language-model system that chooses actions from observations in a loop	Generating a patch in one call
ACI	The abstraction between the agent and computer, including actions, action documentation, environment feedback, and the history passed into the next turn	Merely giving the model a few function names
Environment	The mutable external state formed by the repository, filesystem, terminal, interpreter, and tests	The model internally believing that the code is correct

The full ACI contract: input, state, actions, feedback, and termination

One SWE-agent episode can be reduced to five parts:

Part	Concrete contents	Why it matters
Input	Issue description, initial repository, command documentation, and an optional demonstration	Defines the goal and available operations
Runtime state	Message history, recent observations, current open file, working directory, accumulated edits, and cost or step counters	Grounds the next decision in what has actually happened
Actions	Search, open or scroll through a file, edit a line range, execute programs or tests, and submit	Turns generated intent into repository changes
Environment feedback	Search summaries, numbered code windows, updated content, lint errors, test output, and an explicit no-output message	Gives the model evidence for its next decision
Termination	Explicit submit, cost-limit submission of existing edits, a limit reached with no edits, or early exit after repeated malformed responses	Prevents an unbounded loop and identifies the repository state used for the final patch

flowchart TD
    I["Issue + initial repository state"] --> P["Model chooses the next step"]
    P --> S["Search / navigate"]
    S --> V["Inspect a bounded file window"]
    V --> R["Reproduce the problem and read real output"]
    R --> E["Edit an explicit range"]
    E --> G{"Syntax and edit guardrails pass?"}
    G -->|"No: reject the edit and explain"| P
    G -->|"Yes: show updated code"| T["Run targeted tests and necessary regression checks"]
    T --> Q{"External acceptance passes?"}
    Q -->|"No: preserve failure evidence"| P
    Q -->|"Yes"| D["Inspect diff / remove temporary files"]
    D --> X["submit produces the final patch"]

The two return paths are the important part of the diagram. An edit failure does not become “pretend it worked,” and a test failure does not become “ask the model to believe in the patch again.” The environment converts failure into evidence that the next turn can use.

How the paper designs the action space

SWE-agent’s ACI is built on top of the Linux shell; it does not prohibit ordinary commands. It adds a small set of high-frequency operations designed for a language model.

1. Repository search and navigation

• find_file locates candidates by filename.
• search_file searches for a string inside one file.
• search_dir searches for a string within a directory.

Search does not print an unlimited stream. The paper’s implementation summarizes matches and asks the model to narrow a query when there are too many results. The mechanism is not a magical new search algorithm. It is control over the shape of the observation: enough of a summary to form a global judgment without filling the context window.

2. A bounded file viewer

After open selects a file, the interface displays only a bounded line window at a time; scroll_up, scroll_down, and goto move that window. The observation also includes:

• the full file path;
• the file’s total line count;
• a line number for every displayed line;
• the number of omitted lines above and below the window.

The model therefore knows that it is seeing a slice of a file rather than silently treating a truncated view as the complete source.

3. Precise editing integrated with the viewer

The paper’s edit operation primarily takes a start line, an end line, and replacement text, allowing a multiline replacement in one action. After success, the system automatically shows the updated window. If the integrated linter detects selected errors, the edit is discarded and the agent receives the reason and nearby code.

That creates a small but important transaction semantic:

attempt edit → validate → on success, apply to worktree and show new state
                        → on failure, preserve old state and explain how to retry

A brittle shell edit, by comparison, can leave the file partially broken before the model understands what happened.

4. Context processing is part of the interface

An ACI defines more than commands. SWE-agent also manages command documentation, demonstrations, format errors, and the observation history:

• each turn is asked to produce a judgment and an action, with retries for malformed output;
• a silent command receives an explicit “completed successfully with no output” observation;
• older observations are collapsed into short records while a few recent observations remain detailed;
• post-edit content re-enters the observation so that the next step is less likely to rely on stale code.

A more accurate formula is therefore:

ACI = action space + action documentation + environment-feedback format + guardrails + history-management rules.

A complete run: rejecting a negative timeout

The following trajectory is constructed for teaching; it is not copied from the paper. The issue says that parse_timeout("-1") should raise ValueError, but currently returns -1.

Turn	Agent goal	ACI action	How feedback changes the next step
1	Understand the repository and existing coverage	Inspect status and search for parse_timeout	Returns a small set of implementation and test matches rather than the entire repository
2	Read the implementation in context	open src/config.py, then jump to the match	Shows a numbered window of at most 100 lines and indicates that more content exists
3	Establish that the bug is real	Run a targeted test or minimal Python reproduction	Real output confirms that a negative value is accepted, creating a before-change baseline
4	Discover the project’s existing error semantics	Search adjacent validation code and ValueError tests	Reveals that this project validates the range after parsing rather than silently clamping it
5	Change implementation and test	Apply edit to explicit ranges	A bad indentation is rejected by lint; a corrected edit returns updated source
6	Verify local behavior	Run the new test and existing configuration tests	New and old tests pass, supporting the target fix and a bounded regression claim
7	Verify patch scope	Inspect the diff and check for temporary scripts	Confirms that only implementation and test files changed
8	Terminate	submit	Collects a patch from the final worktree state

Turn 3 is not optional. Without reproduction, the agent may modify the wrong function. Turn 6 also cannot be replaced with “Does the model think this looks correct?” External tests must produce the evidence.

Minimal control-loop pseudocode

state = initialize(issue, repository, command_docs)
 
while state.cost < budget and state.format_errors < error_limit:
    context = build_context(
        issue=state.issue,
        instructions=state.command_docs,
        recent_observations=state.recent_observations,
        compressed_history=state.older_history,
    )
    thought, action = model(context)
 
    if action.name == "submit":
        return collect_patch(state.repository)
 
    result = execute_with_guardrails(action, state.repository)
    state.update(action, result)
 
return collect_patch(state.repository) if state.has_edits else None

This is not presented as a literal reproduction of the implementation. It exposes the minimal causal chain: model weights stay fixed, but changing build_context, the action definitions, or the result representation changes what the model does next.

What actually differs between a custom ACI and a general shell?

Task	A general shell may require	Custom ACI path	Mechanistic benefit
Find a class	A combination of find and grep, possibly with excessive output	One search action with aggregated and bounded results	Less parameter and observation ambiguity
Read a large file	Full cat output or a manually constructed sed -n range	Fixed window, line numbers, and remaining-range metadata	Controls tokens while preserving location
Make a multiline edit	Rewrite the file, use complex sed, or create a temporary script	Replace an explicit line range	Fewer escaping steps and intermediate states
Recover from a bad edit	The command may partly apply and require manual diagnosis	Lint rejects invalid edits and returns a local error	Stops an error from propagating
Interpret no output	Success, failure, and a stalled process may be hard to distinguish immediately	Explicit successful-no-output message	Removes state ambiguity

This does not mean the shell is unprofessional, nor that a modern agent can never become proficient with it. The paper compares particular models, prompts, budgets, and tasks under particular interfaces. The durable engineering claim is that an interface is part of the system, not a neutral pipe for every model.

What the evidence supports—and what it does not

Version 3 evaluates SWE-agent on SWE-bench and HumanEvalFix and ablates search, editing, file-window, and context-processing choices. Key results include:

• SWE-agent with GPT-4 Turbo resolved 12.47% (286) of the then-current 2,294-task full SWE-bench;
• on the 300-task SWE-bench Lite, SWE-agent reached 18.0%, while the shell-only configuration with a demonstration reached 11.0%; the paper’s summary also describes a 10.7-percentage-point gain over a default-shell baseline, so the specific table configuration and demonstration condition matter when interpreting that comparison;
• removing the dedicated editor, stepping through search results one by one, showing entire files, or retaining the complete observation history all performed worse than the selected configuration in the reported ablations;
• the same ACI completed some tasks with Claude 3 Opus, suggesting that the effect was not wholly specific to one base model.

Evidence boundary

These are results from a 2024 paper with the models, benchmark, prompts, cost limits, and implementation available at the time. They support “ACI design affects behavior and pass rate.” They do not show that one command set is optimal for every model, repository, or budget, and they are not a current leaderboard. An ablation changes a concrete interface configuration, so the entire difference cannot automatically be assigned to one isolated detail.

Common failure modes and how to read them

1. Over-broad search: the agent repeatedly consumes irrelevant matches. Narrow scope, aggregate hits, and prompt query refinement.
2. Wrong localization: the agent finds a symbol with the right name but does not trace the actual call path. A reproduction, caller search, and tests must constrain the diagnosis together.
3. Edit loops: the same region changes repeatedly without code execution. This often signals a missing feedback surface or failure to treat tests as observations.
4. Target test passes but a regression appears: local success is not repository acceptance. The interface must expose appropriate regression tests and the final diff.
5. Premature submission: the model substitutes “the code looks plausible” for completion. Termination should depend on external success evidence, not verbal confidence.
6. Too many overlapping tools: several commands express nearly the same action, making tool choice itself a reasoning burden. A better ACI is often smaller and clearer, not endlessly larger.

Historical contribution and today’s engineering interpretation

What the original work established

SWE-agent brought an HCI-style interface question into language-model agents: a model does not touch “the world” transparently through tools. It encounters a representation of possible actions and observations that someone designed. Real-repository evaluation and ablations showed that this representation can change downstream performance without updating model weights.

How to apply the idea today

• Version tool schemas, parameter names, error semantics, and output truncation as part of the evaluated system.
• Define atomic file edits: failure leaves no partial change, while success returns local evidence of the new state.
• Make search, logs, and test output filterable, pageable, or expandable on demand rather than dumping everything by default.
• Record the tuple “model version × prompt × ACI version × environment version,” or regressions will be impossible to attribute.
• Compare interfaces on real tasks instead of asking people to read tool descriptions and declare one subjectively easier.

What the paper does not imply

• It does not imply that custom commands always beat a shell; stronger models, different tasks, or a better shell wrapper can change the result.
• It does not imply that passing tests necessarily satisfies product intent; tests cover only encoded behavior.
• It does not imply that interface work replaces model capability, repository quality, or sound evaluation. ACI is one system layer, not the whole system.

Relationship to Harness Engineering

ACI mainly governs the immediate actions and observations through which an agent uses a computer. Harness Engineering expands the scope to repository knowledge, isolated environments, architecture constraints, CI, observability, merge practices, and continuous cleanup. ACI can be viewed as the layer of a harness closest to each individual action.

Check whether you really understand ACI

1. Why does “the shell can express every operation” fail to refute the value of an ACI?
2. Why is an editor returning “success” insufficient? What state evidence should it return at minimum?
3. If an experiment changes only the search interface while holding the model, tasks, prompt, and budget fixed, what can it answer? What confounders remain?
4. Write a five-step or longer agent trajectory for changing a configuration parser, including at least one failure return path.

Answers and hints

1. Expressiveness says that the task is possible, not that a model can select, compose, and recover from operations reliably under a finite context and budget. ACI targets operability and feedback quality.
2. It should at least report the changed range, updated local content, validation outcome, and whether failure preserved the old state. Tests and diff inspection are still required to validate behavior and scope.
3. It can estimate the search interface’s effect on behavior and performance under those conditions. Randomness, demonstrations, context processing, cost, model version, and task sampling must still be controlled or measured, ideally over repeated runs.
4. A sound answer includes localization → bounded inspection → reproduction → edit plus syntax feedback → targeted tests → necessary regressions → diff inspection → submission. Any failed check returns to diagnosis rather than proceeding to submission.

Previous: SWE-bench · Next: Building Effective Agents · Back to the landmark reading index