SWE-bench: evaluating coding agents with real issues, patches, and tests

Original source

Carlos E. Jimenez et al., ICLR 2024: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. This guide follows arXiv v3.

This is an original learning guide, not a reproduction of the paper. Copyright and the complete work remain with the original source.

What you should be able to do after this guide

By the end, you should be able to:

1. Express a SWE-bench task using problem statement $P$, codebase snapshot $C$, test patch $T$, and reference fix $\delta$.
2. Explain how a real Issue and merged Pull Request are filtered, separated, and execution-validated into a benchmark instance.
3. Define FAIL_TO_PASS and PASS_TO_PASS precisely and explain why every test in both groups must pass.
4. Trace candidate-patch application, test execution, log parsing, and scoring from the base commit.
5. Distinguish four claims: the patch differs from the maintainer’s answer, the patch applies, the tests pass, and the patch is completely correct in real engineering.
6. Explain how repository version, dependency environment, evaluation harness, and data contamination change a result—and why leaderboard numbers require a versioned experimental contract.

Why short function problems are not enough for coding agents

Many traditional code benchmarks provide a function signature and a few tests; the answer may be only several lines. Real repository maintenance is much harder. An Issue describes symptoms using project-specific vocabulary, the defect may live in a different file, and a valid repair may require following call paths, preserving old behavior, matching project conventions, and running tests under historical dependencies.

SWE-bench therefore changes the question from “can a model continue a plausible code snippet?” to:

Given a historical Issue and the codebase as it existed when the problem occurred, produce an applicable patch. Run real tests in a reproducible environment to decide whether it repairs the target behavior without breaking selected existing behavior.

This setup tests more than a base model. Context retrieval, repository navigation, editing interfaces, execution environments, error feedback, and retry policy can all change the resolution rate. That is why SWE-bench later became a major stress test for coding-agent and harness design.

What exactly is in one task?

The paper describes an instance through four core objects:

Symbol	Meaning	Origin	Role during evaluation
$P$	Problem statement	The linked Issue’s title, body, and comments written before the PR’s first commit	Shown to the model to describe the required change
$C$	Codebase snapshot	The original PR’s base_commit	Checked out so the model edits the historical state
$T$	Test patch and related tests	Changes to test files in the original PR diff	Applied by the evaluator to reproduce behavior before and after the fix
$\delta$	Gold/reference patch	Non-test changes in the original PR diff	Used to construct and validate the instance; candidates need not copy it line by line

The model or Agent produces candidate patch $\hat{\delta }$. Scoring cares about the behavior of $\hat{\delta }$ under the tests, not textual similarity to $\delta$. A structurally different implementation can be correct if it satisfies the same executable contract.

The test patch is not an answer sheet for the model

The benchmark’s central task input is $P$ plus $C$; $T$ and $\delta$ belong to construction and evaluation. The paper describes test_patch as unseen test changes used for scoring. A modern evaluation must also state which existing public tests the Agent may execute. “The Agent can run public repository tests” and “the Agent can see hidden scoring tests” are not the same condition.

How GitHub history becomes a benchmark

The complete collection described in arXiv v3 contains 2,294 tasks from 12 popular Python repositories. Those tasks emerged from a three-stage pipeline, not from arbitrarily sampling 2,294 Issues.

Stage 1: select repositories and collect Pull Requests

The authors selected 12 popular open-source repositories written primarily in Python and began with roughly 90,000 Pull Requests. Each PR naturally exposes a base commit, a diff, linked discussion, and timestamps.

Stage 2: filter by repository metadata

A candidate PR had to be merged and also:

1. explicitly link to and resolve a GitHub Issue; and
2. modify test files, indicating that maintainers added an executable check for the problem.

The problem statement aggregates the associated Issue title, body, and comments, but only content posted before the PR’s first commit is included. That temporal cutoff reduces leakage of solution discussion into the task statement.

Stage 3: filter by actual execution

For each candidate, the data pipeline builds a runnable environment and returns to the base commit:

1. Install that historical repository version.
2. Apply test patch $T$ and execute relevant tests to get pre-fix log log_pre.
3. Apply reference fix $\delta$, rerun the same tests, and get post-fix log log_post.
4. Keep the sample only if installation, patch application, and test execution work and at least one target test changes from failure to success.

The original PR diff is divided into $T$ and $\delta$ using test-related paths. This is a scalable engineering heuristic, not perfect semantic understanding of what counts as test code or product code. Dataset quality still depends on repository layout, test coverage, and log-parsing rules.

flowchart TD
    A["Roughly 90,000 real PRs"] --> B{"Linked to and resolves an Issue?"}
    B -->|"No"| R1["Remove"]
    B -->|"Yes"| C{"Contains test changes?"}
    C -->|"No"| R2["Remove"]
    C -->|"Yes"| D["Split test patch T from reference patch δ"]
    D --> E["Check out base commit C and install its environment"]
    E --> F["Apply T only; execute tests for log_pre"]
    F --> G["Apply δ as well; execute tests for log_post"]
    G --> H{"Runnable and at least one fail → pass?"}
    H -->|"No"| R3["Remove"]
    H -->|"Yes"| K["Enter the 2,294-task full collection"]

How two test groups define “resolved”

Comparing each relevant test before and after the reference fix creates status groups. Two of them form SWE-bench’s core scoring contract.

FAIL_TO_PASS: did the target behavior get fixed?

These tests fail with the base commit plus the test patch, then pass after the reference fix is also applied. They encode behavior the Issue asks to change. A candidate must make every FAIL_TO_PASS test pass; otherwise some measured part of the target problem remains unresolved.

PASS_TO_PASS: did the repair preserve existing behavior?

These tests pass before the reference fix and continue to pass afterward. They are not the target defect; they are selected regression protection. A candidate must keep them passing so that it cannot “fix” the new example by deleting validation, hard-coding one input, or breaking an older feature.

The paper’s decision is strict: every named FAIL_TO_PASS and PASS_TO_PASS test must be found in the candidate’s parsed log and have passing status. A missing test or any non-passing status counts as failure.

Candidate outcome	FAIL_TO_PASS	PASS_TO_PASS	Decision
Target defect remains	At least one failure	All pass	Not resolved
New example is fixed but a regression appears	All pass	At least one failure	Not resolved
Test process crashes or target test never runs	Missing/non-passing	Unknown	Not resolved
Target behavior restored and old behavior retained	All pass	All pass	Resolved

FAIL_TO_FAIL and PASS_TO_FAIL are not part of the original paper’s core resolution rule. They may still be diagnostically useful; the benchmark simply chooses the first two groups to define the target and regression contracts.

How a candidate patch is executed and scored

For each $\hat{\delta }$, the evaluator broadly performs these steps:

1. Clean the worktree and check out the instance’s base commit, restoring $C$.
2. Activate the environment for that repository version and install the project.
3. Apply test patch $T$.
4. Apply candidate patch $\hat{\delta }$; the paper’s implementation retries after repairing some formatting defects.
5. Run the test command determined by $T$ and capture the log.
6. Use repository-specific parsing rules to convert the log into a mapping from test name to status.
7. Confirm every required test exists and passes, then return Resolved or Not resolved.

def evaluate(instance, candidate_patch):
    repo = checkout(instance.repo, instance.base_commit)
    env = activate_environment(instance.version)
    install(repo, env)
    apply_patch(repo, instance.test_patch)
 
    if not apply_or_repair_patch(repo, candidate_patch):
        return "not_resolved"
 
    log = run_repository_tests(repo, instance.test_patch)
    status = parse_with_repo_specific_rules(log)
 
    required = instance.FAIL_TO_PASS + instance.PASS_TO_PASS
    if all(status.get(test) == "PASS" for test in required):
        return "resolved"
    return "not_resolved"

Keep the intermediate states distinct. An applicable patch is not necessarily behaviorally correct, and a successfully started test process is not the same as all required tests passing.

Walk through a simplified example

The following fictional pathkit task illustrates the scoring semantics; it is not an actual SWE-bench Issue.

The Issue says safe_join(root, child) incorrectly rejects a valid child path when root ends with a slash. Four existing security tests pass at the base commit, and the PR adds one test reproducing this bug.

Repository state	New target test	Four existing security tests
Base commit + $T$	Fails	All pass
Reference fix $\delta$ also applied	Passes	All pass

The new test is therefore FAIL_TO_PASS; the four older tests are PASS_TO_PASS.

• Candidate A deletes all path-boundary checks. The new test passes, but an old “cannot escape root” test fails, so the task is not resolved.
• Candidate B normalizes the trailing separator while retaining boundary checks. Both groups pass, so it can be resolved even if its code differs from the maintainer’s reference patch.
• Candidate C changes test configuration so the new test is never collected. The target test is missing from the parsed mapping, so evaluation still fails.

Execution-based scoring is closer to behavior than text similarity, but it still covers only the contract represented by the tests.

Why repository version, environment, and containers are part of the score

The same patch can behave differently under different Python, dependency, or system-library versions. The original paper did not merely run everything against the latest repository. It built conda environments for relevant release versions, recorded Python and dependency information, and validated tasks at their base commits.

A modern harness may package this environment in a container, but “we used Docker” is not a complete reproducibility statement. At minimum, an evaluation should lock or record:

• dataset and task-instance version;
• repository mirror and base commit;
• Python version, dependency lock, and system packages;
• container image digest or deterministic build inputs;
• installation command, test command, timeout, and resource limits;
• test patch, log parser, and evaluation-harness version;
• tests, network, tools, and retry budget visible to the Agent.

Two results both labeled “SWE-bench” are not strictly the same experiment if their subset, environment, or harness differs.

How to read the evidence

The 2,294 tasks from 12 Python repositories in paper v3 demonstrate that real Issue–PR history can be converted into a large executable benchmark. Many instances require reasoning across functions, classes, and files rather than completing a self-contained function.

The historical baselines also show how difficult the setup was at the time: with the paper’s specified BM25 retrieval and model snapshot, Claude 2 resolved 1.96% of tasks. That number is useful for understanding the 2023–2024 starting point, not as a current leaderboard statement. Models, agents, benchmark variants, cleaning, and harnesses have all evolved; this guide intentionally does not provide a dynamic ranking.

The more durable conclusion is that real repository repair is bottlenecked not only by code generation, but also by locating relevant context, producing an applicable patch, controlling the execution environment, and reacting to runtime feedback.

Contamination, versioning, and interpretation risks

1. Public history may appear in training or retrieval

Issues, PRs, and reference fixes already existed on GitHub. A model may have encountered them during pretraining, and a networked Agent might retrieve the original PR. Time splits or similar performance across years offer only limited evidence; they cannot prove absence of contamination. Reports must disclose network access, retrieval scope, and model training cutoff when known.

2. Tests are not a complete specification

Passing both groups means the candidate satisfies the instance’s executable scoring contract. It does not guarantee performance, security, style, untested platforms, or every part of user intent.

3. The reference patch is not the only possible answer

$\delta$ is the maintainers’ merged solution and is useful for constructing state transitions and understanding the task. Another smaller or structurally different patch can still be correct. Conversely, retrieving and reproducing $\delta$ may earn a score without demonstrating generalizable software-engineering ability.

4. Benchmark variants cannot be reported interchangeably

Full, Lite, Verified, and later variants differ in filtering and difficulty. A comparison must name the variant, version, instance count, model and Agent configuration, sampling budget, public-test access, and harness.

Common failure modes and a debugging order

Symptom	Common cause	Check first
Candidate patch will not apply	Wrong diff context, path, or line numbers	Correct base commit and patch format
Local tests pass but benchmark fails	Different dependencies, system packages, tests, or harness	Environment image, test patch, and parser versions
All FAIL_TO_PASS pass but instance is unresolved	A PASS_TO_PASS regression or missing target test	Full status of both groups, not only new tests
Model reads much code but edits the wrong file	Retrieval noise or project terminology mismatch	Retrieval recall/ranking and execution feedback
Score is suspiciously high and process is opaque	Historical PR retrieval or test leakage	Network, search, training contamination, and visible files
Same benchmark label cannot be reproduced	Dataset or harness version was omitted	Complete evaluation manifest and raw logs

Historical contribution and today’s engineering interpretation

The paper’s historical contribution was to organize real GitHub Issues, historical code snapshots, maintainer patches, and test transitions into a large execution-based coding benchmark. Whether a task is fixed depends on repository behavior, not textual similarity to an answer.

Today’s engineering interpretation is that coding-agent ability must be accepted through a real execution path. Task definition, code version, environment, tool interface, and feedback harness are all part of what is being measured. A SWE-bench score measures repository repair under a specific contract; it is not complete software-engineering competence and does not replace product-level security or end-to-end acceptance.

Its relation to the next reading is straightforward: SWE-bench defines the problem and scoring contract, while SWE-agent asks what computer interface an Agent should use to inspect and modify the repository.

Check that you can apply the mechanism

1. What are $P$, $C$, $T$, $\delta$, and $\hat{\delta }$? Which primarily help the model solve the task, and which remain on the evaluation side?
2. Why would requiring only FAIL_TO_PASS reward dangerous repairs?
3. Under what conditions should a candidate be resolved even when it differs line by line from the reference patch?
4. How does the original paper derive the two test groups from pre-fix and post-fix logs? Why is a missing target test a failure?
5. When comparing two SWE-bench results, name at least five versions or configuration categories that must be checked.

Reference answers and judgment criteria

1. $P$ is the problem statement, $C$ the base-commit code, $T$ the test patch, $\delta$ the reference fix, and $\hat{\delta }$ the candidate. The model primarily uses $P$ and $C$ to generate a candidate; $T$ and $\delta$ construct and execute the evaluation.
2. A candidate could delete validation, hard-code the new example, or break an old interface while passing the target test. PASS_TO_PASS guards selected existing behavior.
3. If the patch applies and every FAIL_TO_PASS and PASS_TO_PASS test is executed and passes, it satisfies the original scoring contract; textual matching is unnecessary.
4. Run the base commit plus test patch for the pre-fix state, then apply the reference fix for the post-fix state. Tests moving fail→pass and pass→pass form the groups. A missing test provides no evidence that its contract was checked, so it fails.
5. Check at least the benchmark variant and version, instance set, repository and environment, model and Agent configuration, tool/network/test visibility, sampling budget, and evaluation harness.

Previous: MCP launch article · Next: SWE-agent · Back to the landmark reading index