Introduction
Repairing an existing software project usually means understanding the issue, finding the relevant code, making changes, and checking that they satisfy the existing behavioral contract. SWE-bench Lite turns problems of this kind from real open-source projects into tasks: given a repository and an issue, submit a repair patch, then have predetermined tests check the result. This post uses those tasks to compare several ways a large language model can approach code repair.
One approach gives the model the issue and retrieved source code all at once and asks it to output a patch directly. Another puts it into a loop: it decides what to search for next, which file to read, how to edit it, and which checks to run, then continues acting based on the tools’ results. Here, Agent Loop refers to that interaction cycle. The execution framework around the model, responsible for inputs, tools, workspaces, and budgets, is the harness.
These approaches change many conditions at once. The Loop can obtain more source code, write changes into actual files, receive execution results, and call the model repeatedly. Even if its success rate is higher, it is difficult to tell how much each capability contributed. The follow-up experiment therefore added a tool-free multi-turn revision group: it still had only the initial materials, but could revisit and revise its own answers. This provided a direct reference across one-off generation, tool-free revision, and full tool interaction.
The starting point for this work is the previous Agent Harness Lab Retrospective. In that first round, static One-shot solved 4 of 20 tasks, while the full Loop’s task-averaged success rate was 55%, a difference of 35 percentage points. One-shot means delivering an answer after a single call. That result showed the benefit of the whole interaction system as it stood, but did not distinguish the effects of more generation opportunities and new information from the environment.
Round 2 reused the same tasks, improved the repeated-run and comparison design, and then added a static reference with a long output allowance. This post first presents the design and results, then analyzes patch failure stages and three actual trajectories, and finally looks back at the engineering problems encountered while trying to isolate execution capability. Round 2 is complete. The execution ablation reached only part of its preparation, with no formal model runs, and I ultimately decided to stop it. Development, experiments, and verification were carried out in collaboration with AI. The explanations below are retrospective analysis based on the saved inputs, actions, patches, and scoring records.
1. Limitations of the first experiment
1.1 Multiple capabilities changed together
In the first round, referred to below as V1, the static method and Loop differed in source access, editing, execution, and other respects. The static method received the issue and retrieved source snippets, then expressed its changes as a diff. The Loop could continue inspecting the repository, choose actual files, make changes, run commands, and act on the results. How much source it read, how it generated the patch, whether it could inspect execution results, how many times it called the model, and when it stopped all changed together.
V1’s results correspond to the overall difference between those two working systems. For example, the static method might have reasoned out the right algorithm but lacked the file it needed to change. Or it might have identified the file but produced a diff that would not apply. By reading files and editing an actual workspace, the Loop could overcome both obstacles before even using test feedback. Attributing a higher Loop success rate directly to self-correction after running tests would skip these equally concrete differences.
The groups also had different sample counts. V1’s static One-shot ran once per task; the Loop had eight independent runs per task. The 55% was calculated by finding each task’s success proportion and then averaging across tasks. It does not mean that 11 of the 20 tasks succeeded at least once. Repeated samples help reveal stability, but the groups’ estimates of task performance had different precision. The earlier post also discussed coverage from repeated sampling, which is different again from average single-run success probability: succeeding at least once in eight tries does not make every try reliable.
Round 2 first added multi-turn revision without environment interaction, to see what further generation opportunities would produce when the static materials stayed fixed. This comparison still cannot estimate each Loop component’s contribution separately, but it reduces some of the ambiguity introduced by the difference in model-call counts in the first round.
1.2 One-off generation, tool-free revision, and full interaction
An ordinary programming scenario helps explain the three groups. Suppose a configuration-loading function opens a file in text mode by default, while a new parser requires a binary stream. From the issue and function snippet alone, the model might think to add a parameter. Rereading its own answer might also reveal that the parameter never reaches the actual file-opening call. But if the project has wrapper functions, compatibility conventions, or existing tests elsewhere, those appear only when it inspects the repository again.
The first approach reads the supplied materials once and directly returns a patch. The second still has only those original materials, but can revisit and revise its answer. The third can both keep reasoning and actively obtain new source, edit files, and run checks. This example only illustrates the methodological differences; the actual Flask case later shows how these actions occurred in the records.
I label the three groups A0, A2, and B. A0 is a static method with one long output; A2 is revision over at most three calls without environmental feedback; B is the full Loop. Each group has different information conditions. A2 can continue checking its existing answer, but later calls receive no new source code, execution results, or scoring evidence.
information flow in A0, A2, and B. The groups share task materials and a cumulative output cap, but differ in the information available afterward. A0 was a sequential extension added after the A2/B results were available. Official scoring occurs only after candidates are frozen and never enters the model’s revision loop.
Diagram guide: the common initial materials are the issue, task identity, and fixed BM25 source snippets. A0 makes one long request and extracts a diff. A2 appends its own prior answers and follows a frozen revision prompt for at most 3 calls, selecting a candidate diff mechanically. B chooses actions, reads/searches/edits/executes, receives real tool results, and collects the final working-tree diff within 60 steps. All candidates are frozen before independent official scoring. A2 updates a candidate only for an untruncated answer containing an extractable, nonempty diff.
To read the charts, click an image, choose its original size, and scroll within it. This is the recommended way to inspect labels and notes on a phone.
2. Round 2 design and variable control
2.1 Comparing A2/B first, then adding A0
Round 2 reused 20 tasks from 10 repositories, with four independent runs per task per group. Each group therefore planned 80 runs, or 240 across all three. Each run started from its own initial state; an answer or tool result from one run was never passed to another run on the same task. The records number the four repeats 0–3, and I retain that numbering below. The four repeats estimate stability on the same task; they are not one Agent repairing the same issue four times in succession.
A2 and B finished first. Their runs were interleaved, and they shared the fixed issue and static retrieval materials. Retrieval used BM25, a method that selects snippets by term matching and document relevance. It supplied roughly 13K tokens of source context; tokens are the basic units used to measure text for a model. These snippets were not guaranteed to cover every relevant file. Later A2 calls could see only the existing input and the model’s own earlier answers, with no new source, execution results, or scoring feedback. B could additionally search, read, and edit the actual repository and run commands.
A0 was added after A2/B already had results. It reused the system prompt and input from A2‘s first call and froze its own protocol before requests for this group began. This kept the initial materials for A0 and A2 identical and supplied a direct reference for one long output. But the sequence cannot be rewritten: the three groups were not a randomized comparison preregistered at the same point and completed concurrently. Adding A0 narrowed the research question as the work progressed, and that design fact limits the causal interpretation the results can carry.
The model recorded for this round was deepseek-flash, corresponding in the materials available at the time to V4.1 Flash. The configuration enabled reasoning mode and requested high reasoning effort. V1 used a different model version, so this post compares only A0/A2/B within Round 2. I do not subtract results across rounds to calculate a benefit from upgrading the framework.
2.2 Output budgets and candidate selection
All three groups had a cumulative completion budget cap of 131,072 tokens. This included reasoning tokens; they must not be added again. A0 could use the full cap in one call. A2 and B had per-call caps of 65,536, further constrained by their remaining allowance. A2 allowed at most three calls and B at most 60 steps. Neither had to use the entire budget.
Although the groups shared a cumulative output cap, their actual computational-resource consumption still differed. B repeatedly carried context forward, received tool output, and occupied execution time; later A2 calls had to reread its own answers. An equal output cap constrains one budget dimension. It does not establish “same compute.”
Candidate selection also has to be kept separate from results. A2 did not score three answers and choose the best. It kept the last eligible candidate in generation order: the answer had to be untruncated by length and contain an extractable, nonempty diff. Only an eligible extracted patch replaced the previous candidate; a nonempty explanation did not count as a new patch. B instead collected actual changes from the final workspace. All groups froze candidates before independent scoring, and the model could not see that round’s official result.
These rules affect what can fail. If a later A2 answer only explained its reasoning without supplying an extractable patch, that explanation could not be treated as a better candidate. If B wrote a test file into the workspace, that file could also enter the final diff. Final scoring received the collected patch bytes, so files left behind during generation could affect the result. The Flask case below shows a concrete example.
3. Results across the three groups and differences between tasks
3.1 The full 20 tasks and the common 19-task set
| Comparison set | A0 |
A2 |
B |
|---|---|---|---|
| Full 20 tasks, 80 runs each | 25/80 · 31.25% | Two nulls; no mean reported on the same basis | 54/80 · 67.50% |
| Common 19 tasks, 76 runs each | 25/76 · 32.89% | 31/76 · 40.79% | 52/76 · 68.42% |
Across the full 20 tasks, A0 succeeded in 25/80 runs, for a task-averaged success rate of 31.25%; B succeeded in 54/80, or 67.50%. B was 36.25 percentage points higher than A0. Because each task had four runs, the task mean equals total successes divided by 80. The statistical unit is still the task; the 80 runs are not 80 unrelated software problems.
The four runs of a task share its issue, repository, and difficulty. Treating them all as separate observations would overstate the information provided by repetition. Confidence intervals therefore used a paired bootstrap by task: each resample drew a set of tasks, retained all four repeats of each selected task, and used the same task indices in the compared groups. Bootstrap measures variation in observed results by resampling. The pairing was at task level; runs with the same repeat number across groups did not share random numbers.
The frozen analysis used 10,000 resamples, seed 20261005, and the original task set. For the full 20 tasks, the 95% interval for B−A0 was 18.75–53.75 percentage points. This interval describes uncertainty on the current task set; it does not automatically generalize into a guarantee for all repositories, issues, or models. With only 20 tasks, which difficult tasks a resample includes can noticeably change the difference.
Comparing all three groups requires handling A2‘s two nulls. Both were runs of requests-2148. Null means the frozen scoring pipeline produced no numeric result; it cannot be filled in as a failed score of 0. To compare the groups on the same set of complete tasks, the original analysis used a common 19-task set, excluding requests-2148 as a whole. This differs from dropping only the two missing values and calculating 31/78, which would give that task less weight in A2 while leaving its full weight in the other groups.
The table retains the frozen reviewed layer: scoring records reviewed under the predetermined evidence conditions. Repeat 2 of A0 on requests-2148 was originally null too. An existing review assigned it 0 on complete evidence of patch application failure. The two A2 cases lack the complete evidence needed for the same determination, so they remain null. A single error message is not enough to fill in their results.
On the common 19 tasks, A2−A0 was 7.89 percentage points, with a 95% interval of 2.63–14.47. B−A2 was 27.63 points, with an interval of 9.21–47.37; B−A0 was 35.53 points, with an interval of 17.11–53.95.
frozen task means and task-paired 95% bootstrap intervals. On the full 20 tasks, A0/B each have 80 runs. On the common 19 tasks, all three groups have 76 runs, excluding the whole requests-2148 task containing two A2 nulls. A0 was added later; the comparisons do not imply that all three groups were designed concurrently.
Chart guide: the left panel is the primary full-20-task A0/B comparison; the right is the supplementary common-19-task comparison across three groups. The horizontal axes show task-averaged success rate (%), with equal weight per task. The bottom annotations show differences in percentage points and their paired intervals. No full-20-task A2 point estimate is plotted.
3.2 Group differences, output volume, and request costs
On the common 19 tasks, tool-free multi-turn revision group A2 had a higher task mean than one-off generation group A0. This is a difference in their final candidates’ performance. The records do not establish from it that any particular revision changed an incorrect answer into a correct one. A0 and A2 differed simultaneously in call count, per-call budget, and revision structure, and were completed sequentially. The 7.89-percentage-point difference cannot be directly attributed to any one self-revision mechanism.
The B versus A2 difference comes closer to the original research question: even after giving the static method further opportunities to revise, the full interactive system still had a higher task mean. Yet B‘s additional capabilities remained bundled. Reading more source can improve localization, editing actual files can improve patch delivery, execution can expose behavioral errors, and feedback can influence later searches and changes. For this task set and the recorded model configuration, the full Loop had a higher task-averaged success rate; the contributions of source interaction, actual editing, execution, feedback, and revision still cannot be estimated separately.
Actual output volume helps assess that explanation too. Across the 80 final run chains, cumulative completion tokens were approximately 2.23 million for A0, 5.07 million for A2, and 2.21 million for B, averaging about 27,855, 63,378, and 27,563 per run. A2 produced about 2.3 times as much output as A0, while B was close to A0. These records do not show a relationship in which longer output means higher success. This measures recorded output volume, excludes tool execution costs, and is not a measurement of the model’s actual computation.
Using historical prices and token records for all known formal requests, estimated costs were about US6.21 for A2, and US
Task-level results show that B‘s advantage was uneven. Across the full 20 tasks, B was higher than A0 on 12, tied on 7, and lower on 1. Across the common 19, B was higher than A2 on 8, tied on 10, and lower on 1. The average gain was concentrated in some tasks, rather than a small improvement on every task.
differences in successful runs per task, with four repeats per task; one additional success equals 25 percentage points. B−A0 covers the full 20 tasks, while B−A2 covers the common 19. Tasks are ordered by B−A0. The figure shows variation across tasks and does not treat repeat numbers as paired random samples across groups.
Chart guide: horizontal positions are differences in success counts, not percentages. Red means B is higher, gray a tie, and orange lower. The requests-2148 row has no B−A2 value because two A2 results remain null.
The four tasks with the most concentrated gains were django-14238, pytest-11143, pytest-7490, and sphinx-11445. A0/A2 were both 0/4 and B was 4/4 on all four. These tasks alone account for 16 of the net 21 additional successes in B−A2 on the common set. Next, I return to the trajectories behind high-gain tasks and failures.
Counterexamples are equally concrete. On seaborn-3190, A0 scored 3/4 and A2 4/4, but B only 2/4. On flask-4045, xarray-4248, and sphinx-7686, all three groups scored 0/4. The full Loop can still miss constraints, choose ineffective actions, or rely on local checks with insufficient coverage. I first classify failures along the patch-processing path, then return to individual trajectories.
4. Failure stages in patch generation, application, and scoring
4.1 Candidate delivery and stage classification
A model answer must pass through candidate extraction, patch application, testing, and other steps before receiving a score. First there must be an extractable candidate. It must meet diff-format requirements and apply to the source in the scoring environment. Only then come official test construction, test execution, and result aggregation. A final 0 might mean no patch was delivered at the very beginning, or that some functionality was correctly repaired but later behavioral checks failed. The location of failure therefore provides one clue to differences in success rate.
The failure-stage analysis after the formal experiment recorded the earliest confirmable failure in each run. Empty candidate (E) means no usable final diff, even if the answer contained an explanation. Format failure (M) means the patch structure could not be parsed normally. Application failure (P) means the format was valid, but the changes could not be applied to the scoring source. Test or scoring failure (T) covers test-patch construction, test execution, and result determination after application. Success is R, while question marks retain nulls without numeric scores.
This classification locates the layer of failure, but is not a complete root-cause explanation for every run. In particular, T does not directly mean the model’s product code was wrong. If the official test patch cannot coexist with the candidate, the tests may never have been fully constructed. If only some tests run, the final score can still fail even when all those tests pass. Explaining those cases requires returning to trajectories and logs.
the path from candidate generation to scoring. E/M/P identify earlier delivery gates; T combines test and scoring failures after application, so the label alone does not establish a root cause in product code. Local execution during generation and official scoring after candidate freeze are separate stages.
Diagram guide: generate an answer or edited working tree → extract/collect the candidate → mechanical format check → apply it to the scoring baseline → construct official tests by injecting the test patch → collect and run tests → produce the official report and score. E is empty, M is malformed, P is rejected at application, T is applied but not officially resolved, and R is officially resolved. Missing results or insufficient stage evidence remain ?/null; one observed error cannot establish a 0.
Across each group’s 80 planned run slots, E/M/P/T/R counts were 5/13/16/21/25 for A0; 3/5/17/22/31 for A2, plus two nulls; and 3/0/0/23/54 for B. All 77 nonempty B candidates passed format and application checks. This is consistent with how B delivered changes: the model edited actual files, then the workspace diff was collected, reducing the burden of hand-writing patch syntax and context.
But those counts do not explain all of B‘s gain. More early failures in A0/A2 and fewer in B show that delivery paths genuinely differed. Candidates that passed application were already different selected subsets in each group. Comparing their conditional success rates cannot establish that execution feedback accounts for the remaining difference. One capability can also improve several stages at once: newly read source can both prevent application failure and help the model choose the correct repair location.
earliest confirmable stage across 80 planned slots per group. The two A2 nulls are retained. All 77 nonempty B candidates passed format and application checks; three others were empty. T combines test- or scoring-stage failures and does not represent a single kind of product-code error.
Chart guide: the horizontal axis counts planned slots (80 per group). Light gray E = empty candidate; blue M = invalid format; ochre P = application failure; orange T = applied but not officially resolved; green R = officially resolved; purple ? = insufficient evidence / null.
4.2 Gaps in static materials and rejected patch applications
The previous figure counts 33 application failures by final stage. Application logs record one further rejection whose final score was missing and remains an A2 null, so this section discusses 34 rejection records. In 29 of them, a file to be modified was absent from the fixed BM25 materials. The model might know which concept needed changing while lacking the file’s exact code and context. Being able to keep reading files offers a way to turn such guesses into confirmation.
In other application rejections, the relevant code was already visible. In repeat 1 of A0 on flask-4992, the relevant old code was in the input, but the candidate still used a hunk with no context, which the initial git apply rejected. A hunk is a block in a diff describing a local change. Without surrounding unchanged code, an applicator cannot confirm the location in the normal way. Later partial applications and reverse-patch hints occurred after the first rejection; they cannot be read backward as evidence that the scoring environment had already fixed the issue at the start.
In several other rejections, the old code blocks could not be found even in B‘s fully visible files. Some scoring environments had also undergone preparation commits and were not at the original public task’s starting commit. The available materials are insufficient to confirm the complete scoring source baseline and its effects for every run. The 29 material gaps are therefore an observable fact, not a final root-cause verdict on all 34 failures—and certainly not a basis for assigning an exact percentage-point contribution to dynamic source reading.
For the scoring pipeline, a repair idea must become an applicable change to the target code. B‘s actual edits and diff collection reduced that delivery burden. This observation still does not estimate how many successes that feature independently produced.
5. Execution feedback and failure causes in actual trajectories
5.1 Django-14238: a type fix and adjustments to the reproduction environment
django-14238 concerns configuration of the default primary-key type. Django’s DEFAULT_AUTO_FIELD specifies the field class used when a model does not explicitly declare a primary key. In this task, a user defined MyBigAutoField, inheriting from BigAutoField, and wanted to make it the default. The framework rejected it as not being a subclass of AutoField.
The detail lies in type-compatibility logic. The automatic primary-key fields do not form a simple, ordinary inheritance chain. Django uses special checks in AutoFieldMeta so that BigAutoField and SmallAutoField can also be treated as AutoField. The original check listed accepted classes but recognized only those classes themselves, not their derived classes. Built-in BigAutoField therefore passed, while custom MyBigAutoField(BigAutoField) fell outside the check. The core change replaced a membership test with an inheritance check:
1 | - return subclass in self._subclasses or super().__subclasscheck__(subclass) |
This accepted subclasses of the listed types while retaining the other normal type-checking path.
On this task, A0/A2 were both 0/4 and B was 4/4. The trajectories show more of what happened within those four successes. Repeat 0 of B checked relevant issubclass results, then ran default-primary-key tests, with 10 passing checks recorded. Model-options tests also recorded 22 passes and 3 skips. These checks connected the type relationships in the code with user-visible configuration, rather than merely checking patch syntax.
In repeat 3, execution feedback was mainly used to adjust the reproduction environment. The product-code change was complete at step 5. Later, the custom reproduction failed to import at step 9, encountered an assertion failure at step 10, inspected the difference between global and app configuration at step 11, and finally passed the temporary myapp end-to-end reproduction at step 12. The product patch did not change after step 5. What kept changing were the validation script and Django app configuration.
repeat 3 of B on django-14238. The product fix was complete at step 5. Later failures prompted adjustments to the reproduction environment until the end-to-end check passed at step 12; the product patch did not change again.
Diagram guide: step 5 changes the subclass check to issubclass(...); step 9 fails to import the temporary app because the contenttypes module is missing; step 10 finds the actual primary key still using AutoField; step 11 diagnoses an app-level default_auto_field still pointing to AutoField; step 12 adjusts myapp and reaches E2E OK. Green marks a passing temporary reproduction. The final diff remains the original one-line product change and excludes the temporary app files; the archived official result is 1.
Failure feedback in this run pointed to the reproduction environment; the subsequent adjustments did not revise the product patch again. Import errors, assertion results, and configuration checks helped the model locate problems in the temporary app’s setup. Looking only at failure followed by success would miss what was actually changed. This trajectory validated an existing product change; it does not establish that several rounds of product-logic correction were needed to fix it.
Some static candidates also proposed a similar repair direction but did not succeed at stages such as patch delivery. This task therefore supports the conclusion that B completed it more reliably through its combination of localization, actual editing, and checks. It does not separately prove that removing execution would turn all four B successes into failures. That was the question the later execution ablation sought to ask, requiring another experiment that genuinely controlled static-access conditions.
5.2 Flask-4992: local tests conflicting with official test construction
flask-4992 is the configuration-loading example introduced earlier. Flask’s Config.from_file opens a file and passes the file object to the user-supplied load function. Default text mode suits some parsers, but Python’s tomllib.load requires a binary file object. A suitable interface needs to let callers select the opening mode so the parser receives the stream type it requires, while keeping default calls compatible.
The B records include successful paths. In repeat 1, for example, one local configuration test recorded 18 passes and another temporary simulation recorded 2. Official scoring then ran 19 tests and succeeded. Different test counts are unsurprising: generation-time checks were selected by the model, while official scoring also constructed the regression tests specified by the task. Local checks assist generation; official tests make the final determination. A difference in counts is no reason to combine them into one validation run.
Repeat 0 left the opposite example. At step 8, the model added tests/static/config.toml for local validation. Local configuration tests at step 9 recorded 19 passes. It later checked a broader suite and compared with the original code, made no further product-logic changes, and ended generation at step 15. The candidate added text/binary mode selection, but the test fixture remained in the workspace and entered the final candidate with the diff.
A fixture is fixed data used by tests. The problem was that the official test patch also needed to add the same path. Candidate application itself succeeded, but official test construction then encountered a same-name file conflict, so the planned test patch could not be fully applied. The final log recorded 18 passes, while the frozen score remained 0. This failure was in test construction and the scoring pipeline. The 19 local passes cannot retroactively make it a success; nor can we ignore the conflict and conclude that the new API implementation must have been wrong.
repeat 0 of B on flask-4992. A local fixture was submitted with the candidate, creating a same-path conflict during official test construction after generation ended. The 19 local passes and the final 0 belong to different stages.
Diagram guide: during generation, step 8 adds config.toml and changes test_config.py; step 9 records 19 local passes; step 15 finishes and freezes the candidate. After generation, candidate application succeeds, the official test patch collides with existing tests/static/config.toml, the archived run collects 18 tests with 18 passes, and the official report still records reward = 0. That later official information is not returned to the model.
This record contains both generation-time local feedback and the official result after generation. The model actually received the output of local tests it ran itself. Final official scoring happened after it finished. It did not know the test patch that would be applied and had no opportunity to act on the eventual conflict. When discussing Agent self-revision, we need to specify which checks supplied the feedback and whether it appeared before or after candidate freeze.
This also exposes a delivery risk within the interactive system itself. A real workspace makes modification and validation convenient, but not every file created during validation necessarily belongs in the product candidate. There is an interface between workspace state, candidate collection, and test construction. Mishandling that interface can make a candidate fail before it reaches the intended tests.
5.3 Flask-4045: a missed exception contract in an adjacent API
Not every failure can be explained by the environment or test construction. All three groups scored 0/4 on flask-4045, and repeat 3 of B shows an ordinary but important omission. This issue concerns nested Blueprint names. A Blueprint organizes a set of routes. When Blueprints are nested, the framework joins parent and child names with dots, such as parent.child.c. The dots express hierarchy; allowing users to put dots into an individual blueprint’s name would confuse that meaning.
The existing code also prohibited dots in endpoint. An endpoint is the identifier used to look up a route in reverse, not a URL path; it is a separate entry point from the blueprint’s name. The original issue mentioned the existing endpoint restriction and requested a corresponding restriction on blueprint names. The official tests also checked how the error was exposed: an endpoint containing a dot should raise ValueError, but the existing path used assert and actually raised AssertionError.
This B run added a name check when constructing a blueprint, validated illegal names, and demonstrated valid nesting. Those local checks passed, but did not cover the API path bp.route('/', endpoint='a.b'). The official log explicitly failed there: the old assertion in add_url_rule still raised AssertionError, not the exception type the test expected.
Finding this omission depends on which contracts the checks cover. Name composition, constructor parameters, route registration, and exception types can each be right or wrong independently. The model chose an apparently reasonable reproduction path but missed the adjacent entry point. Passing local tests covers only the checks that were executed. Untested API paths and exception contracts may still determine the final score.
These three cases show different uses and limitations of execution results. Django’s feedback helped set up the validation environment correctly. Local passes for Flask configuration loading did not ensure that the official tests could be constructed. Flask’s naming checks left a genuine product-behavior gap. All involve execution results, but those results played different roles in each trajectory. Aggregate success rates do not directly reveal those process differences; trajectory analysis adds that layer.
6. Execution ablation design and engineering blockers
6.1 Controlling static access and editing conditions
After Round 2, one question remained: with static access and editing conditions held equal, how much additional benefit would program execution provide? One could imagine disabling B‘s Shell and comparing the result. But the old Shell did more than run tests: cat could read files, rg could search, and commands could help establish workspace state. Removing that tool would also remove static-access capabilities, again changing several conditions together.
The proposed execution ablation therefore designed FULL and STATIC. An ablation removes a capability under controlled conditions to observe the change in outcome. Both groups would keep the same trusted static interfaces for reading, searching, and editing source, with the same inputs and candidate collection. FULL would additionally receive model-callable run_task and its execution results; STATIC would have no program execution. Both would still receive reading or editing errors, so STATIC would not be a long-output method with no feedback at all.
Each execution would read a source snapshot from the moment of invocation. Side effects such as generated files would not be written directly back into the candidate. Thus, FULL would additionally obtain execution information while both groups edited deliverable code through the same channels. The plan was four runs per task per group on the original 20 tasks: 160 formal runs. It sought to estimate the combined increment from controlled program execution and execution information, given equal static access and actual editing.
planned FULL/STATIC groups share static access, editing, and candidate collection; only FULL adds controlled execution and execution results. A valid comparison still required acceptance checks for source-to-execution binding on actual tasks, network admission, and transfer to scoring. The study has stopped; formal generation and scoring are both at 0/160.
Diagram guide: the same trusted initial source feeds identical structured read/search/edit interfaces. STATIC retains reading/editing feedback without program execution. FULL adds fixed execution templates and isolated results, with no execution side effects written into the candidate. Shared candidate collection is followed by independent official scoring. Outstanding requirements are trustworthy source and runtime consistency, real DNS/network admission, and a reliable candidate-to-scoring bridge. The 154 offline engineering checks are not additional SWE-bench successes.
This comparison never entered formal runs. The difficulty was first fixing the source each task should use, confirming that execution loaded the version just edited, and getting network-dependent tests and final scoring to work end to end. If any link was missing, the feedback FULL received might not correspond to its current candidate. The following two tasks exposed incomplete work in source binding and networked execution.
6.2 Astropy: binding source provenance to compiled artifacts
Astropy made one runtime-environment problem concrete. It contains extensions that need compilation. For pure Python files, checking import paths is usually fairly direct. With extension modules, the source on disk, binary artifacts generated during preparation, and modules actually loaded by tests may be at different layers. Even after the source is edited, old compiled artifacts can still load and produce passing results unrelated to the current candidate. Comparing execution capability first requires determining which source each task should use, then confirming that both preparation and runtime environments come from that source.
Existing component-acceptance records included successful preparation and imports, but full source capture was rejected at a 0777 permission check on cextern/trim_wcslib.sh, and the subsequent program execution did not occur. Permission 0777 allows every user to read, write, and execute the file; the original capture entry point could not directly treat such a writable path as trusted source. The block happened before execution, so it says nothing about the correctness of a model patch. Changing the permission number alone would not prove that the source came from the right task either.
Later read-only evidence confirmed that the permission existed before preparation, but did not establish its cause. Hashing preexisting source can only establish that two sets of bytes match; by itself, it cannot prove that they are the authoritative input for that task. The new full provenance path still lacked a file inventory fixed in advance from another trusted source. A newly captured inventory cannot prove the correctness of its own capture. This is the distinction between provenance validation and content consistency.
The minimal approach also needed to resolve execution binding after edits: which artifacts could be reused, which required rebuilding, and which modules the tests actually loaded. Existing offline tests could check cache rules and boundaries on simulated filesystems. They could not prove that actual C/Cython extensions had been correctly rebuilt, linked, and loaded for the current edits. Successful preparation, successful import, and trustworthy execution after editing are three different completion conditions.
These conditions directly affect the comparison’s validity. If FULL receives results from an old binary, its feedback does not correspond to the current edit. Before formal runs, the relationship among source, compiled artifacts, and loaded modules has to be verified on actual tasks. Records of successful preparation or imports alone do not complete that acceptance check.
6.3 requests: network-dependent tests and transfer to scoring
requests-2148 concerns how a low-level socket reset is wrapped as a requests exception. Environment acceptance encountered a different layer of difficulty: existing regression tests needed to reach HTTPBin and check how Authorization was handled on redirects between hosts, so they required a genuinely usable networked test path.
The old process required the remaining validity of relevant DNS results to meet a fixed preflight window. In one record, the remaining TTL was about 100.84 seconds, below the required 130 seconds, so the process stopped before starting the official verifier. TTL is how long a DNS record may still be cached. This shows only that the admission condition was not met on that attempt; it does not show that the requests patch or tests were incorrect.
The later design reconfirmed allowed addresses when a connection was used, reducing dependence on remaining TTL at a one-off preflight. Offline simulations could validate expiry judgments and error paths, but actual HTTPS fetching and the controlled forwarding channel were not connected yet. Part of the interface logic was complete; real regression tests were still not running.
Beyond networking, there was a final gap in scoring. Candidates generated in the new execution environment had to reach the existing scoring environment reliably, with raw logs and verdicts returned afterward. The minimal approach reused the official parser, bound frozen candidates to reports, and checked how missing states were represented. But a mock scorer producing a correctly formatted report does not establish that real transfer, result collection, and cleanup work end to end. Across 20 tasks, the requirement was a reliable connection between complete inputs, actual execution, and the scoring pipeline.
6.4 Incomplete acceptance checks and the decision to stop
I later tried reducing the implementation scope. The old records called this engineering approach minimal B; it was not an additional experimental group beyond Round 2’s B. It did not require migrating the existing Windows/WSL scorer as a whole, did not use a broad DNS preflight as the sole admission gate, and reused the scoring parser. These choices reduced design dependencies, but did not remove the work needed for a valid experiment: a trusted source inventory, binding preparation and build artifacts, a real network channel, reliable candidate delivery and report retrieval, and cleanup of runtime state still needed to be finished and integrated.
The final offline Linux engineering-check report recorded 154 tests: 153 passed, 1 was skipped because it depended on Windows, and there were no failures or errors. That number answers whether the new code’s interfaces and boundaries held on synthetic filesystems, fake DNS, and mock scorers. It is not 154 SWE-bench tasks or formal FULL/STATIC samples. The old stage had only partial component acceptance for a few tasks; that cannot become a guarantee that the new approach runs across all 20.
The investment needed to complete the ablation lay mainly in connecting actual environments, verifying provenance, and end-to-end acceptance. Model requests were only part of the cost. Existing API cost estimates could not substitute for judging that engineering work.
Ultimately, I chose to stop the execution ablation. Formal FULL/STATIC generation and scoring are both at 0/160, with no estimate of execution capability’s gain. Stopping was a choice about further investment, not an observation that execution feedback was ineffective.
The wrap-up retained the existing results, incomplete acceptance checks, and remaining questions. Round 2’s success-rate differences still stand, while the comparison needed to separate execution’s contribution remains unfinished.
7. Conclusions and mechanisms left unverified
Round 2’s clearest finding is that the full Loop had a higher task-averaged success rate on this fixed task set and recorded model configuration. A0‘s one long output, A2‘s revision without environmental feedback, and B‘s environmental interaction are three different ways of working. A2 had a higher mean than A0, and B a higher mean than A2. Those differences occurred between systems with different call structures and information conditions. The existing results do not support treating them as causal contributions from individual capabilities.
Tool calls in these trajectories changed the available information or workspace state. Reading a file can change what the model knows; editing real files changes the state to be delivered; execution turns some assumptions into observable results; subsequent actions determine how those results are used. The current experiment supports better performance by the complete combination on this round’s tasks. Each capability’s independent contribution still requires other comparisons.
Interpreting an execution result also requires knowing which check it corresponds to. Django’s failure messages came from the reproduction setup. Flask’s green output covered only local checks. Official scoring could additionally encounter conflicts with candidate files during test construction. An Agent needs not just a result, but also to know which code state, assertions, and stage it corresponds to. Those conditions determine how broad a judgment a result can support.
This suggests a straightforward lesson for my everyday coding: I’ll prioritize letting the model inspect real code, deliver actual changes, and verify behavior relevant to the issue, while choosing the check coverage and runtime environment carefully. A local test pass is one piece of evidence; completion of the whole task is another judgment. For repositories depending on compiled artifacts, external services, or complex initialization, the environment itself also needs to be inspectable.
This project did not answer how much execution capability contributes under equal static access. That question remains open after FULL/STATIC stopped. The existing results support a higher task-averaged success rate for the full Loop on this round’s fixed tasks; they cannot separate the contributions of source interaction, execution, feedback, and revision. The three cases preserve the concrete processes: which checks helped validate changes, which local passes failed to cover the whole task, and how delivery interfaces affected final results. These records can inform later practice while retaining questions that require another valid comparison to answer.