Repairing an existing software project usually means understanding the issue, finding the relevant code, making changes, and checking that they satisfy the existing behavioral contract. SWE-bench Lite turns problems of this kind from real open-source projects into tasks: given a repository and an issue, submit a repair patch, then have predetermined tests check the result. This post uses those tasks to compare several ways a large language model can approach code repair.

One approach gives the model the issue and retrieved source code all at once and asks it to output a patch directly. Another puts it into a loop: it decides what to search for next, which file to read, how to edit it, and which checks to run, then continues acting based on the tools’ results. Here, Agent Loop refers to that interaction cycle. The execution framework around the model, responsible for inputs, tools, workspaces, and budgets, is the harness.

These approaches change many conditions at once. The Loop can obtain more source code, write changes into actual files, receive execution results, and call the model repeatedly. Even if its success rate is higher, it is difficult to tell how much each capability contributed. The follow-up experiment therefore added a tool-free multi-turn revision group: it still had only the initial materials, but could revisit and revise its own answers. This provided a direct reference across one-off generation, tool-free revision, and full tool interaction.

The starting point for this work is the previous Agent Harness Lab Retrospective. In that first round, static One-shot solved 4 of 20 tasks, while the full Loop’s task-averaged success rate was 55%, a difference of 35 percentage points. One-shot means delivering an answer after a single call. That result showed the benefit of the whole interaction system as it stood, but did not distinguish the effects of more generation opportunities and new information from the environment.

Round 2 reused the same tasks, improved the repeated-run and comparison design, and then added a static reference with a long output allowance. This post first presents the design and results, then analyzes patch failure stages and three actual trajectories, and finally looks back at the engineering problems encountered while trying to isolate execution capability. Round 2 is complete. The execution ablation reached only part of its preparation, with no formal model runs, and I ultimately decided to stop it. Development, experiments, and verification were carried out in collaboration with AI. The explanations below are retrospective analysis based on the saved inputs, actions, patches, and scoring records.

1. Limitations of the first experiment

1.1 Multiple capabilities changed together

In the first round, referred to below as V1, the static method and Loop differed in source access, editing, execution, and other respects. The static method received the issue and retrieved source snippets, then expressed its changes as a diff. The Loop could continue inspecting the repository, choose actual files, make changes, run commands, and act on the results. How much source it read, how it generated the patch, whether it could inspect execution results, how many times it called the model, and when it stopped all changed together.

V1’s results correspond to the overall difference between those two working systems. For example, the static method might have reasoned out the right algorithm but lacked the file it needed to change. Or it might have identified the file but produced a diff that would not apply. By reading files and editing an actual workspace, the Loop could overcome both obstacles before even using test feedback. Attributing a higher Loop success rate directly to self-correction after running tests would skip these equally concrete differences.

The groups also had different sample counts. V1’s static One-shot ran once per task; the Loop had eight independent runs per task. The 55% was calculated by finding each task’s success proportion and then averaging across tasks. It does not mean that 11 of the 20 tasks succeeded at least once. Repeated samples help reveal stability, but the groups’ estimates of task performance had different precision. The earlier post also discussed coverage from repeated sampling, which is different again from average single-run success probability: succeeding at least once in eight tries does not make every try reliable.

Round 2 first added multi-turn revision without environment interaction, to see what further generation opportunities would produce when the static materials stayed fixed. This comparison still cannot estimate each Loop component’s contribution separately, but it reduces some of the ambiguity introduced by the difference in model-call counts in the first round.

1.2 One-off generation, tool-free revision, and full interaction

An ordinary programming scenario helps explain the three groups. Suppose a configuration-loading function opens a file in text mode by default, while a new parser requires a binary stream. From the issue and function snippet alone, the model might think to add a parameter. Rereading its own answer might also reveal that the parameter never reaches the actual file-opening call. But if the project has wrapper functions, compatibility conventions, or existing tests elsewhere, those appear only when it inspects the repository again.

The first approach reads the supplied materials once and directly returns a patch. The second still has only those original materials, but can revisit and revise its answer. The third can both keep reasoning and actively obtain new source, edit files, and run checks. This example only illustrates the methodological differences; the actual Flask case later shows how these actions occurred in the records.

I label the three groups A0, A2, and B. A0 is a static method with one long output; A2 is revision over at most three calls without environmental feedback; B is the full Loop. Each group has different information conditions. A2 can continue checking its existing answer, but later calls receive no new source code, execution results, or scoring evidence.

Diagram guide: the common initial materials are the issue, task identity, and fixed BM25 source snippets. A0 makes one long request and extracts a diff. A2 appends its own prior answers and follows a frozen revision prompt for at most 3 calls, selecting a candidate diff mechanically. B chooses actions, reads/searches/edits/executes, receives real tool results, and collects the final working-tree diff within 60 steps. All candidates are frozen before independent official scoring. A2 updates a candidate only for an untruncated answer containing an extractable, nonempty diff.

To read the charts, click an image, choose its original size, and scroll within it. This is the recommended way to inspect labels and notes on a phone.

2. Round 2 design and variable control

2.1 Comparing A2/B first, then adding A0

Round 2 reused 20 tasks from 10 repositories, with four independent runs per task per group. Each group therefore planned 80 runs, or 240 across all three. Each run started from its own initial state; an answer or tool result from one run was never passed to another run on the same task. The records number the four repeats 0–3, and I retain that numbering below. The four repeats estimate stability on the same task; they are not one Agent repairing the same issue four times in succession.

A2 and B finished first. Their runs were interleaved, and they shared the fixed issue and static retrieval materials. Retrieval used BM25, a method that selects snippets by term matching and document relevance. It supplied roughly 13K tokens of source context; tokens are the basic units used to measure text for a model. These snippets were not guaranteed to cover every relevant file. Later A2 calls could see only the existing input and the model’s own earlier answers, with no new source, execution results, or scoring feedback. B could additionally search, read, and edit the actual repository and run commands.

A0 was added after A2/B already had results. It reused the system prompt and input from A2‘s first call and froze its own protocol before requests for this group began. This kept the initial materials for A0 and A2 identical and supplied a direct reference for one long output. But the sequence cannot be rewritten: the three groups were not a randomized comparison preregistered at the same point and completed concurrently. Adding A0 narrowed the research question as the work progressed, and that design fact limits the causal interpretation the results can carry.

The model recorded for this round was deepseek-flash, corresponding in the materials available at the time to V4.1 Flash. The configuration enabled reasoning mode and requested high reasoning effort. V1 used a different model version, so this post compares only A0/A2/B within Round 2. I do not subtract results across rounds to calculate a benefit from upgrading the framework.

2.2 Output budgets and candidate selection

All three groups had a cumulative completion budget cap of 131,072 tokens. This included reasoning tokens; they must not be added again. A0 could use the full cap in one call. A2 and B had per-call caps of 65,536, further constrained by their remaining allowance. A2 allowed at most three calls and B at most 60 steps. Neither had to use the entire budget.

Although the groups shared a cumulative output cap, their actual computational-resource consumption still differed. B repeatedly carried context forward, received tool output, and occupied execution time; later A2 calls had to reread its own answers. An equal output cap constrains one budget dimension. It does not establish “same compute.”

Candidate selection also has to be kept separate from results. A2 did not score three answers and choose the best. It kept the last eligible candidate in generation order: the answer had to be untruncated by length and contain an extractable, nonempty diff. Only an eligible extracted patch replaced the previous candidate; a nonempty explanation did not count as a new patch. B instead collected actual changes from the final workspace. All groups froze candidates before independent scoring, and the model could not see that round’s official result.

These rules affect what can fail. If a later A2 answer only explained its reasoning without supplying an extractable patch, that explanation could not be treated as a better candidate. If B wrote a test file into the workspace, that file could also enter the final diff. Final scoring received the collected patch bytes, so files left behind during generation could affect the result. The Flask case below shows a concrete example.

3. Results across the three groups and differences between tasks

3.1 The full 20 tasks and the common 19-task set

Across the full 20 tasks, A0 succeeded in 25/80 runs, for a task-averaged success rate of 31.25%; B succeeded in 54/80, or 67.50%. B was 36.25 percentage points higher than A0. Because each task had four runs, the task mean equals total successes divided by 80. The statistical unit is still the task; the 80 runs are not 80 unrelated software problems.

The four runs of a task share its issue, repository, and difficulty. Treating them all as separate observations would overstate the information provided by repetition. Confidence intervals therefore used a paired bootstrap by task: each resample drew a set of tasks, retained all four repeats of each selected task, and used the same task indices in the compared groups. Bootstrap measures variation in observed results by resampling. The pairing was at task level; runs with the same repeat number across groups did not share random numbers.

The frozen analysis used 10,000 resamples, seed 20261005, and the original task set. For the full 20 tasks, the 95% interval for B−A0 was 18.75–53.75 percentage points. This interval describes uncertainty on the current task set; it does not automatically generalize into a guarantee for all repositories, issues, or models. With only 20 tasks, which difficult tasks a resample includes can noticeably change the difference.

Comparing all three groups requires handling A2‘s two nulls. Both were runs of requests-2148. Null means the frozen scoring pipeline produced no numeric result; it cannot be filled in as a failed score of 0. To compare the groups on the same set of complete tasks, the original analysis used a common 19-task set, excluding requests-2148 as a whole. This differs from dropping only the two missing values and calculating 31/78, which would give that task less weight in A2 while leaving its full weight in the other groups.

The table retains the frozen reviewed layer: scoring records reviewed under the predetermined evidence conditions. Repeat 2 of A0 on requests-2148 was originally null too. An existing review assigned it 0 on complete evidence of patch application failure. The two A2 cases lack the complete evidence needed for the same determination, so they remain null. A single error message is not enough to fill in their results.

On the common 19 tasks, A2−A0 was 7.89 percentage points, with a 95% interval of 2.63–14.47. B−A2 was 27.63 points, with an interval of 9.21–47.37; B−A0 was 35.53 points, with an interval of 17.11–53.95.

Chart guide: the left panel is the primary full-20-task A0/B comparison; the right is the supplementary common-19-task comparison across three groups. The horizontal axes show task-averaged success rate (%), with equal weight per task. The bottom annotations show differences in percentage points and their paired intervals. No full-20-task A2 point estimate is plotted.

3.2 Group differences, output volume, and request costs

On the common 19 tasks, tool-free multi-turn revision group A2 had a higher task mean than one-off generation group A0. This is a difference in their final candidates’ performance. The records do not establish from it that any particular revision changed an incorrect answer into a correct one. A0 and A2 differed simultaneously in call count, per-call budget, and revision structure, and were completed sequentially. The 7.89-percentage-point difference cannot be directly attributed to any one self-revision mechanism.

The B versus A2 difference comes closer to the original research question: even after giving the static method further opportunities to revise, the full interactive system still had a higher task mean. Yet B‘s additional capabilities remained bundled. Reading more source can improve localization, editing actual files can improve patch delivery, execution can expose behavioral errors, and feedback can influence later searches and changes. For this task set and the recorded model configuration, the full Loop had a higher task-averaged success rate; the contributions of source interaction, actual editing, execution, feedback, and revision still cannot be estimated separately.

Actual output volume helps assess that explanation too. Across the 80 final run chains, cumulative completion tokens were approximately 2.23 million for A0, 5.07 million for A2, and 2.21 million for B, averaging about 27,855, 63,378, and 27,563 per run. A2 produced about 2.3 times as much output as A0, while B was close to A0. These records do not show a relationship in which longer output means higher success. This measures recorded output volume, excludes tool execution costs, and is not a measurement of the model’s actual computation.

Using historical prices and token records for all known formal requests, estimated costs were about US$2.69 for `A0`, US$6.21 for A2, and US$4.17 for `B`; another four `B` requests lacked records that could be priced. The `B` cost includes interrupted earlier attempts, so it covers a different set from the output totals for the final 80 run chains above. These amounts are estimates for known model requests, not complete bills, and exclude tool execution and time spent setting up and maintaining environments. Judging whether an approach is worth its cost also requires considering those inputs alongside its benefits on the specific task.$
Task-level results show that B‘s advantage was uneven. Across the full 20 tasks, B was higher than A0 on 12, tied on 7, and lower on 1. Across the common 19, B was higher than A2 on 8, tied on 10, and lower on 1. The average gain was concentrated in some tasks, rather than a small improvement on every task.

Chart guide: horizontal positions are differences in success counts, not percentages. Red means B is higher, gray a tie, and orange lower. The requests-2148 row has no B−A2 value because two A2 results remain null.

The four tasks with the most concentrated gains were django-14238, pytest-11143, pytest-7490, and sphinx-11445. A0/A2 were both 0/4 and B was 4/4 on all four. These tasks alone account for 16 of the net 21 additional successes in B−A2 on the common set. Next, I return to the trajectories behind high-gain tasks and failures.

Counterexamples are equally concrete. On seaborn-3190, A0 scored 3/4 and A2 4/4, but B only 2/4. On flask-4045, xarray-4248, and sphinx-7686, all three groups scored 0/4. The full Loop can still miss constraints, choose ineffective actions, or rely on local checks with insufficient coverage. I first classify failures along the patch-processing path, then return to individual trajectories.

4. Failure stages in patch generation, application, and scoring

4.1 Candidate delivery and stage classification

A model answer must pass through candidate extraction, patch application, testing, and other steps before receiving a score. First there must be an extractable candidate. It must meet diff-format requirements and apply to the source in the scoring environment. Only then come official test construction, test execution, and result aggregation. A final 0 might mean no patch was delivered at the very beginning, or that some functionality was correctly repaired but later behavioral checks failed. The location of failure therefore provides one clue to differences in success rate.

The failure-stage analysis after the formal experiment recorded the earliest confirmable failure in each run. Empty candidate (E) means no usable final diff, even if the answer contained an explanation. Format failure (M) means the patch structure could not be parsed normally. Application failure (P) means the format was valid, but the changes could not be applied to the scoring source. Test or scoring failure (T) covers test-patch construction, test execution, and result determination after application. Success is R, while question marks retain nulls without numeric scores.

This classification locates the layer of failure, but is not a complete root-cause explanation for every run. In particular, T does not directly mean the model’s product code was wrong. If the official test patch cannot coexist with the candidate, the tests may never have been fully constructed. If only some tests run, the final score can still fail even when all those tests pass. Explaining those cases requires returning to trajectories and logs.

Diagram guide: generate an answer or edited working tree → extract/collect the candidate → mechanical format check → apply it to the scoring baseline → construct official tests by injecting the test patch → collect and run tests → produce the official report and score. E is empty, M is malformed, P is rejected at application, T is applied but not officially resolved, and R is officially resolved. Missing results or insufficient stage evidence remain ?/null; one observed error cannot establish a 0.

Across each group’s 80 planned run slots, E/M/P/T/R counts were 5/13/16/21/25 for A0; 3/5/17/22/31 for A2, plus two nulls; and 3/0/0/23/54 for B. All 77 nonempty B candidates passed format and application checks. This is consistent with how B delivered changes: the model edited actual files, then the workspace diff was collected, reducing the burden of hand-writing patch syntax and context.

But those counts do not explain all of B‘s gain. More early failures in A0/A2 and fewer in B show that delivery paths genuinely differed. Candidates that passed application were already different selected subsets in each group. Comparing their conditional success rates cannot establish that execution feedback accounts for the remaining difference. One capability can also improve several stages at once: newly read source can both prevent application failure and help the model choose the correct repair location.

Chart guide: the horizontal axis counts planned slots (80 per group). Light gray E = empty candidate; blue M = invalid format; ochre P = application failure; orange T = applied but not officially resolved; green R = officially resolved; purple ? = insufficient evidence / null.

4.2 Gaps in static materials and rejected patch applications

The previous figure counts 33 application failures by final stage. Application logs record one further rejection whose final score was missing and remains an A2 null, so this section discusses 34 rejection records. In 29 of them, a file to be modified was absent from the fixed BM25 materials. The model might know which concept needed changing while lacking the file’s exact code and context. Being able to keep reading files offers a way to turn such guesses into confirmation.

In other application rejections, the relevant code was already visible. In repeat 1 of A0 on flask-4992, the relevant old code was in the input, but the candidate still used a hunk with no context, which the initial git apply rejected. A hunk is a block in a diff describing a local change. Without surrounding unchanged code, an applicator cannot confirm the location in the normal way. Later partial applications and reverse-patch hints occurred after the first rejection; they cannot be read backward as evidence that the scoring environment had already fixed the issue at the start.

In several other rejections, the old code blocks could not be found even in B‘s fully visible files. Some scoring environments had also undergone preparation commits and were not at the original public task’s starting commit. The available materials are insufficient to confirm the complete scoring source baseline and its effects for every run. The 29 material gaps are therefore an observable fact, not a final root-cause verdict on all 34 failures—and certainly not a basis for assigning an exact percentage-point contribution to dynamic source reading.

For the scoring pipeline, a repair idea must become an applicable change to the target code. B‘s actual edits and diff collection reduced that delivery burden. This observation still does not estimate how many successes that feature independently produced.

5. Execution feedback and failure causes in actual trajectories

5.1 Django-14238: a type fix and adjustments to the reproduction environment

django-14238 concerns configuration of the default primary-key type. Django’s DEFAULT_AUTO_FIELD specifies the field class used when a model does not explicitly declare a primary key. In this task, a user defined MyBigAutoField, inheriting from BigAutoField, and wanted to make it the default. The framework rejected it as not being a subclass of AutoField.

The detail lies in type-compatibility logic. The automatic primary-key fields do not form a simple, ordinary inheritance chain. Django uses special checks in AutoFieldMeta so that BigAutoField and SmallAutoField can also be treated as AutoField. The original check listed accepted classes but recognized only those classes themselves, not their derived classes. Built-in BigAutoField therefore passed, while custom MyBigAutoField(BigAutoField) fell outside the check. The core change replaced a membership test with an inheritance check:

1
2

- return subclass in self._subclasses or super().__subclasscheck__(subclass)
+ return issubclass(subclass, self._subclasses) or super().__subclasscheck__(subclass)

This accepted subclasses of the listed types while retaining the other normal type-checking path.

On this task, A0/A2 were both 0/4 and B was 4/4. The trajectories show more of what happened within those four successes. Repeat 0 of B checked relevant issubclass results, then ran default-primary-key tests, with 10 passing checks recorded. Model-options tests also recorded 22 passes and 3 skips. These checks connected the type relationships in the code with user-visible configuration, rather than merely checking patch syntax.

In repeat 3, execution feedback was mainly used to adjust the reproduction environment. The product-code change was complete at step 5. Later, the custom reproduction failed to import at step 9, encountered an assertion failure at step 10, inspected the difference between global and app configuration at step 11, and finally passed the temporary myapp end-to-end reproduction at step 12. The product patch did not change after step 5. What kept changing were the validation script and Django app configuration.

Diagram guide: step 5 changes the subclass check to issubclass(...); step 9 fails to import the temporary app because the contenttypes module is missing; step 10 finds the actual primary key still using AutoField; step 11 diagnoses an app-level default_auto_field still pointing to AutoField; step 12 adjusts myapp and reaches E2E OK. Green marks a passing temporary reproduction. The final diff remains the original one-line product change and excludes the temporary app files; the archived official result is 1.

Failure feedback in this run pointed to the reproduction environment; the subsequent adjustments did not revise the product patch again. Import errors, assertion results, and configuration checks helped the model locate problems in the temporary app’s setup. Looking only at failure followed by success would miss what was actually changed. This trajectory validated an existing product change; it does not establish that several rounds of product-logic correction were needed to fix it.

Some static candidates also proposed a similar repair direction but did not succeed at stages such as patch delivery. This task therefore supports the conclusion that B completed it more reliably through its combination of localization, actual editing, and checks. It does not separately prove that removing execution would turn all four B successes into failures. That was the question the later execution ablation sought to ask, requiring another experiment that genuinely controlled static-access conditions.

5.2 Flask-4992: local tests conflicting with official test construction

flask-4992 is the configuration-loading example introduced earlier. Flask’s Config.from_file opens a file and passes the file object to the user-supplied load function. Default text mode suits some parsers, but Python’s tomllib.load requires a binary file object. A suitable interface needs to let callers select the opening mode so the parser receives the stream type it requires, while keeping default calls compatible.

The B records include successful paths. In repeat 1, for example, one local configuration test recorded 18 passes and another temporary simulation recorded 2. Official scoring then ran 19 tests and succeeded. Different test counts are unsurprising: generation-time checks were selected by the model, while official scoring also constructed the regression tests specified by the task. Local checks assist generation; official tests make the final determination. A difference in counts is no reason to combine them into one validation run.

Repeat 0 left the opposite example. At step 8, the model added tests/static/config.toml for local validation. Local configuration tests at step 9 recorded 19 passes. It later checked a broader suite and compared with the original code, made no further product-logic changes, and ended generation at step 15. The candidate added text/binary mode selection, but the test fixture remained in the workspace and entered the final candidate with the diff.

A fixture is fixed data used by tests. The problem was that the official test patch also needed to add the same path. Candidate application itself succeeded, but official test construction then encountered a same-name file conflict, so the planned test patch could not be fully applied. The final log recorded 18 passes, while the frozen score remained 0. This failure was in test construction and the scoring pipeline. The 19 local passes cannot retroactively make it a success; nor can we ignore the conflict and conclude that the new API implementation must have been wrong.

Diagram guide: during generation, step 8 adds config.toml and changes test_config.py; step 9 records 19 local passes; step 15 finishes and freezes the candidate. After generation, candidate application succeeds, the official test patch collides with existing tests/static/config.toml, the archived run collects 18 tests with 18 passes, and the official report still records reward = 0. That later official information is not returned to the model.

This record contains both generation-time local feedback and the official result after generation. The model actually received the output of local tests it ran itself. Final official scoring happened after it finished. It did not know the test patch that would be applied and had no opportunity to act on the eventual conflict. When discussing Agent self-revision, we need to specify which checks supplied the feedback and whether it appeared before or after candidate freeze.

This also exposes a delivery risk within the interactive system itself. A real workspace makes modification and validation convenient, but not every file created during validation necessarily belongs in the product candidate. There is an interface between workspace state, candidate collection, and test construction. Mishandling that interface can make a candidate fail before it reaches the intended tests.

5.3 Flask-4045: a missed exception contract in an adjacent API

Not every failure can be explained by the environment or test construction. All three groups scored 0/4 on flask-4045, and repeat 3 of B shows an ordinary but important omission. This issue concerns nested Blueprint names. A Blueprint organizes a set of routes. When Blueprints are nested, the framework joins parent and child names with dots, such as parent.child.c. The dots express hierarchy; allowing users to put dots into an individual blueprint’s name would confuse that meaning.

The existing code also prohibited dots in endpoint. An endpoint is the identifier used to look up a route in reverse, not a URL path; it is a separate entry point from the blueprint’s name. The original issue mentioned the existing endpoint restriction and requested a corresponding restriction on blueprint names. The official tests also checked how the error was exposed: an endpoint containing a dot should raise ValueError, but the existing path used assert and actually raised AssertionError.

This B run added a name check when constructing a blueprint, validated illegal names, and demonstrated valid nesting. Those local checks passed, but did not cover the API path bp.route('/', endpoint='a.b'). The official log explicitly failed there: the old assertion in add_url_rule still raised AssertionError, not the exception type the test expected.

Finding this omission depends on which contracts the checks cover. Name composition, constructor parameters, route registration, and exception types can each be right or wrong independently. The model chose an apparently reasonable reproduction path but missed the adjacent entry point. Passing local tests covers only the checks that were executed. Untested API paths and exception contracts may still determine the final score.

These three cases show different uses and limitations of execution results. Django’s feedback helped set up the validation environment correctly. Local passes for Flask configuration loading did not ensure that the official tests could be constructed. Flask’s naming checks left a genuine product-behavior gap. All involve execution results, but those results played different roles in each trajectory. Aggregate success rates do not directly reveal those process differences; trajectory analysis adds that layer.

6. Execution ablation design and engineering blockers

6.1 Controlling static access and editing conditions

After Round 2, one question remained: with static access and editing conditions held equal, how much additional benefit would program execution provide? One could imagine disabling B‘s Shell and comparing the result. But the old Shell did more than run tests: cat could read files, rg could search, and commands could help establish workspace state. Removing that tool would also remove static-access capabilities, again changing several conditions together.

The proposed execution ablation therefore designed FULL and STATIC. An ablation removes a capability under controlled conditions to observe the change in outcome. Both groups would keep the same trusted static interfaces for reading, searching, and editing source, with the same inputs and candidate collection. FULL would additionally receive model-callable run_task and its execution results; STATIC would have no program execution. Both would still receive reading or editing errors, so STATIC would not be a long-output method with no feedback at all.

Each execution would read a source snapshot from the moment of invocation. Side effects such as generated files would not be written directly back into the candidate. Thus, FULL would additionally obtain execution information while both groups edited deliverable code through the same channels. The plan was four runs per task per group on the original 20 tasks: 160 formal runs. It sought to estimate the combined increment from controlled program execution and execution information, given equal static access and actual editing.

Diagram guide: the same trusted initial source feeds identical structured read/search/edit interfaces. STATIC retains reading/editing feedback without program execution. FULL adds fixed execution templates and isolated results, with no execution side effects written into the candidate. Shared candidate collection is followed by independent official scoring. Outstanding requirements are trustworthy source and runtime consistency, real DNS/network admission, and a reliable candidate-to-scoring bridge. The 154 offline engineering checks are not additional SWE-bench successes.

This comparison never entered formal runs. The difficulty was first fixing the source each task should use, confirming that execution loaded the version just edited, and getting network-dependent tests and final scoring to work end to end. If any link was missing, the feedback FULL received might not correspond to its current candidate. The following two tasks exposed incomplete work in source binding and networked execution.

6.2 Astropy: binding source provenance to compiled artifacts

Astropy made one runtime-environment problem concrete. It contains extensions that need compilation. For pure Python files, checking import paths is usually fairly direct. With extension modules, the source on disk, binary artifacts generated during preparation, and modules actually loaded by tests may be at different layers. Even after the source is edited, old compiled artifacts can still load and produce passing results unrelated to the current candidate. Comparing execution capability first requires determining which source each task should use, then confirming that both preparation and runtime environments come from that source.

Existing component-acceptance records included successful preparation and imports, but full source capture was rejected at a 0777 permission check on cextern/trim_wcslib.sh, and the subsequent program execution did not occur. Permission 0777 allows every user to read, write, and execute the file; the original capture entry point could not directly treat such a writable path as trusted source. The block happened before execution, so it says nothing about the correctness of a model patch. Changing the permission number alone would not prove that the source came from the right task either.

Later read-only evidence confirmed that the permission existed before preparation, but did not establish its cause. Hashing preexisting source can only establish that two sets of bytes match; by itself, it cannot prove that they are the authoritative input for that task. The new full provenance path still lacked a file inventory fixed in advance from another trusted source. A newly captured inventory cannot prove the correctness of its own capture. This is the distinction between provenance validation and content consistency.

The minimal approach also needed to resolve execution binding after edits: which artifacts could be reused, which required rebuilding, and which modules the tests actually loaded. Existing offline tests could check cache rules and boundaries on simulated filesystems. They could not prove that actual C/Cython extensions had been correctly rebuilt, linked, and loaded for the current edits. Successful preparation, successful import, and trustworthy execution after editing are three different completion conditions.

These conditions directly affect the comparison’s validity. If FULL receives results from an old binary, its feedback does not correspond to the current edit. Before formal runs, the relationship among source, compiled artifacts, and loaded modules has to be verified on actual tasks. Records of successful preparation or imports alone do not complete that acceptance check.

6.3 requests: network-dependent tests and transfer to scoring

requests-2148 concerns how a low-level socket reset is wrapped as a requests exception. Environment acceptance encountered a different layer of difficulty: existing regression tests needed to reach HTTPBin and check how Authorization was handled on redirects between hosts, so they required a genuinely usable networked test path.

The old process required the remaining validity of relevant DNS results to meet a fixed preflight window. In one record, the remaining TTL was about 100.84 seconds, below the required 130 seconds, so the process stopped before starting the official verifier. TTL is how long a DNS record may still be cached. This shows only that the admission condition was not met on that attempt; it does not show that the requests patch or tests were incorrect.

The later design reconfirmed allowed addresses when a connection was used, reducing dependence on remaining TTL at a one-off preflight. Offline simulations could validate expiry judgments and error paths, but actual HTTPS fetching and the controlled forwarding channel were not connected yet. Part of the interface logic was complete; real regression tests were still not running.

Beyond networking, there was a final gap in scoring. Candidates generated in the new execution environment had to reach the existing scoring environment reliably, with raw logs and verdicts returned afterward. The minimal approach reused the official parser, bound frozen candidates to reports, and checked how missing states were represented. But a mock scorer producing a correctly formatted report does not establish that real transfer, result collection, and cleanup work end to end. Across 20 tasks, the requirement was a reliable connection between complete inputs, actual execution, and the scoring pipeline.

6.4 Incomplete acceptance checks and the decision to stop

I later tried reducing the implementation scope. The old records called this engineering approach minimal B; it was not an additional experimental group beyond Round 2’s B. It did not require migrating the existing Windows/WSL scorer as a whole, did not use a broad DNS preflight as the sole admission gate, and reused the scoring parser. These choices reduced design dependencies, but did not remove the work needed for a valid experiment: a trusted source inventory, binding preparation and build artifacts, a real network channel, reliable candidate delivery and report retrieval, and cleanup of runtime state still needed to be finished and integrated.

The final offline Linux engineering-check report recorded 154 tests: 153 passed, 1 was skipped because it depended on Windows, and there were no failures or errors. That number answers whether the new code’s interfaces and boundaries held on synthetic filesystems, fake DNS, and mock scorers. It is not 154 SWE-bench tasks or formal FULL/STATIC samples. The old stage had only partial component acceptance for a few tasks; that cannot become a guarantee that the new approach runs across all 20.

The investment needed to complete the ablation lay mainly in connecting actual environments, verifying provenance, and end-to-end acceptance. Model requests were only part of the cost. Existing API cost estimates could not substitute for judging that engineering work.

Ultimately, I chose to stop the execution ablation. Formal FULL/STATIC generation and scoring are both at 0/160, with no estimate of execution capability’s gain. Stopping was a choice about further investment, not an observation that execution feedback was ineffective.

The wrap-up retained the existing results, incomplete acceptance checks, and remaining questions. Round 2’s success-rate differences still stand, while the comparison needed to separate execution’s contribution remains unfinished.

7. Conclusions and mechanisms left unverified

Round 2’s clearest finding is that the full Loop had a higher task-averaged success rate on this fixed task set and recorded model configuration. A0‘s one long output, A2‘s revision without environmental feedback, and B‘s environmental interaction are three different ways of working. A2 had a higher mean than A0, and B a higher mean than A2. Those differences occurred between systems with different call structures and information conditions. The existing results do not support treating them as causal contributions from individual capabilities.

Tool calls in these trajectories changed the available information or workspace state. Reading a file can change what the model knows; editing real files changes the state to be delivered; execution turns some assumptions into observable results; subsequent actions determine how those results are used. The current experiment supports better performance by the complete combination on this round’s tasks. Each capability’s independent contribution still requires other comparisons.

Interpreting an execution result also requires knowing which check it corresponds to. Django’s failure messages came from the reproduction setup. Flask’s green output covered only local checks. Official scoring could additionally encounter conflicts with candidate files during test construction. An Agent needs not just a result, but also to know which code state, assertions, and stage it corresponds to. Those conditions determine how broad a judgment a result can support.

This suggests a straightforward lesson for my everyday coding: I’ll prioritize letting the model inspect real code, deliver actual changes, and verify behavior relevant to the issue, while choosing the check coverage and runtime environment carefully. A local test pass is one piece of evidence; completion of the whole task is another judgment. For repositories depending on compiled artifacts, external services, or complex initialization, the environment itself also needs to be inspectable.

This project did not answer how much execution capability contributes under equal static access. That question remains open after FULL/STATIC stopped. The existing results support a higher task-averaged success rate for the full Loop on this round’s fixed tasks; they cannot separate the contributions of source interaction, execution, feedback, and revision. The three cases preserve the concrete processes: which checks helped validate changes, which local passes failed to cover the whole task, and how delivery interfaces affected final results. These records can inform later practice while retaining questions that require another valid comparison to answer.

Lately, I kept coming across discussions about ChatGPT getting dumber. One explanation was that a node’s unclean IP could make GPT perform worse.

I already used ChatGPT and Codex every day—for coding, studying, and projects. I’d become pretty dependent on them. Seeing that claim, it was hard not to care at all.

As it happened, a couple of days earlier I’d also run the pelican riding a bicycle test in both the web version and Codex. The prompt was this:

Create an HTML page containing a 2D animation of a pelican riding a bicycle, drawn with SVG.

HTML describes a web page’s structure, while SVG can describe lines and shapes directly in code. I wanted to see whether the model could use code alone to draw a pelican riding a bicycle and then animate it.

These are the two results I saved. The first came from Codex, the second from the ChatGPT web version.

Codex GPT-6 Astra High

ChatGPT web version, GPT-6 Pro

My immediate impression was that the web version was more to my taste. The design, details, and overall finish of the page made the difference feel quite noticeable.

But the models, runtime environments, and ways of using tools weren’t identical, and I hadn’t repeated the test while changing only the IP. A difference in output doesn’t mean the IP caused it.

As for my existing proxy, I’d been using OuO: 130 yuan a year for 100 GB a month. Its everyday speed was fine, and the Japanese nodes had reasonably low latency.

Still, the exit was shared with other people. I looked through another round of node checks, and the more I looked, the less settled I felt.

So I thought: since AI was already a tool I used every day, why not give it a separate exit that was fixed, dedicated, and low-risk? I could pay a little more if it meant I could stop thinking about it.

I thought buying a new proxy would be the end of it.

Instead, I went from tinkering with the computer to tinkering with the phone, and ended up building a proxy chain.

Buying a dedicated IP should be simple enough, right?

At the start, I really didn’t have much systematic knowledge of proxies. Normally, I’d import a subscription, pick a node, and flip the switch. The smaller the number under the node, the happier I was.

But once I was actually ready to pay for a clean IP, I discovered that the terms on the product page meant quite different things.

Dedicated means whether the exit is shared with other proxy customers. Static means whether the address changes. Residential / ISP describes the network’s origin or ownership. Speed and stability are separate questions again.

ISP stands for Internet Service Provider. For a first approximation, a Dedicated ISP Proxy is a dedicated proxy using ISP-sourced addresses. Decodo’s explanation of static residential / ISP proxies also combines ISP sourcing with server hosting; it doesn’t mean a household has set aside its fiber modem just for me.

Once I broke it all down, I really wanted two things: a clean IP and a stable connection. More specifically, a fixed exit, preferably used only by me, that actually worked.

I eventually chose Decodo’s monthly dedicated plan with three IPs: two in Japan and one in the US. Japan would be my daily primary and backup; I’d keep the US one for situations that genuinely needed a US exit later.

The allocation page when I bought the IPs.

Screenshot guide: two Japanese allocations in Tokyo and one US allocation in New York City. The columns are IP, country, city, and ASN; the actual IP addresses are hidden.

After receiving them, I checked each one on Scamalytics, AbuseIPDB, and IPinfo. IPQualityScore required registration, so I didn’t go any further with it.

At the time of the checks, all three IPs had Scamalytics risk scores of 0 and 0 reports on AbuseIPDB. The US IP appeared as a Comcast ISP network. Some databases classified the two Japanese exits as Hosting, meaning server-hosting networks.

At first, I was bothered that the Japanese exits were classified as server hosting. Then I thought about it: a pretty-looking check report wasn’t really what I wanted. If they were dedicated, fixed, and stable in actual use, there was no need to chase the labels too hard.

These were only third-party checks, after all. No abuse reports doesn’t mean an address has never been abused, and it doesn’t mean OpenAI will necessarily accept that exit.

After the purchase, the dashboard gave me three proxy entry points. The same domain, with different ports, corresponded to my three assigned exits. I confirmed them with actual requests and gave them these names:

The entry address and the final exit IP are different things. The client connects to Decodo’s entry point; the target website sees the address used when the request leaves Decodo.

At this point, I thought I was halfway done.

The difficult part was only just beginning.

The IP checks looked great. The websites I wanted wouldn’t open.

I first added the three nodes to FlClash as HTTP proxies.

The dashboard showed a Japanese exit, and sometimes the nodes even returned latency results. So I opened Google, YouTube, and ChatGPT.

Then the pages started spinning.

Google wouldn’t open. YouTube wouldn’t open. ChatGPT seemed to work at first, then stopped as well. I had to switch back to OuO to keep asking ChatGPT what was going on.

That was a little awkward.

I’d bought a new proxy to use AI more reliably. Now even troubleshooting the new proxy depended on the old one.

To avoid blaming everything on the client, I started testing with curl. Curl is a command-line network request tool: you can explicitly specify the proxy and target URL, then print the response or error.

In Windows PowerShell, I used curl.exe to specify the proxy and target explicitly. The command below replaces my real username with a placeholder. When only a username is supplied, curl prompts for the password, so it doesn’t need to go into command history.

1

curl.exe -I --proxy "http://isp.decodo.com:10003" --proxy-user "YOUR_USERNAME" "https://www.google.com/"

The result was still the same:

1

curl: (56) Recv failure: Connection was reset

Neither Japanese port worked, and switching to the US port gave the same result.

Yet Decodo’s own checking endpoint returned a response.

My first reaction was: does this product only support checking itself, rather than actually browsing the web?

Before I understood the proxy, PowerShell tripped me up twice

Once, I copied curl -U ... into PowerShell and got this instead:

1

Invoke-WebRequest：无法处理参数，因为参数名称 U 具有二义性。

That error says Invoke-WebRequest couldn’t process the parameter because the name U was ambiguous. My environment treated curl as an alias for Invoke-WebRequest; it hadn’t called the real curl at all. I had to change it to curl.exe before I could continue.

Later, the AI gave me another command using ^ for line continuation. I pasted it in, and PowerShell complained again. That was CMD’s continuation syntax, not PowerShell’s.

At that point, I decided to keep subsequent short commands on one line wherever possible.

These errors had absolutely nothing to do with proxy quality. But they mixed into the troubleshooting and made everything look messier than it really was.

Support said it was supported, so I tested one variable at a time

I contacted Decodo’s live support.

An automated support agent answered first. It explicitly said Google and ChatGPT were supported and YouTube should work too, and asked me to try another ISP port.

The support response I received at the time.

I tested all three ports again. Still resets.

Support then asked me to try a few unrelated HTTPS websites to see whether only those targets were affected. HTTPS adds TLS encryption to HTTP; TLS stands for Transport Layer Security and protects connection contents while verifying the server’s identity.

This time, there was a difference.

GitHub returned 301 or 200, Cloudflare returned 200, and ordinary HTTPS websites opened too.

A successful response looked roughly like this:

1
2
3

HTTP/1.1 200 Connection established

HTTP/1.1 200 OK

The proxy wasn’t completely unreachable. Different targets behaved differently.

I put the rounds of testing into a table:

After changing devices and trying the hotspot, the AI and I briefly turned our attention to the provider’s account or gateway.

But these tests still couldn’t fully rule out network factors. Changing the access network doesn’t mean changing every shared part of the network path. The devices and phone hotspot could still traverse similar upstream networks. That narrowed the search; it didn’t establish that Decodo was responsible.

Along the way, support also asked me to check background VPNs, firewalls, and antivirus software. I explained the tests I’d already done again and was eventually transferred to a human. The human support agent couldn’t reproduce the error with the same request on their side, so they asked me to try cellular data. The laptop/hotspot comparison in the table came from that part of the investigation.

There was another small trap: once, I used -I with ip.decodo.com/json and got 405 Method Not Allowed. This time, the network hadn’t broken. The endpoint accepted only GET / OPTIONS, while I’d sent a HEAD request to retrieve headers only. Removing -I restored the original GET check.

An HTTP error response and a dropped connection need to be judged separately.

curl -v finally showed me where the connection stopped

The human support agent asked for a verbose log, meaning a more detailed connection trace printed with -v.

This was the test command. The account name and authentication information in the logs have been replaced with placeholders.

1

curl.exe -v -o NUL --proxy "http://isp.decodo.com:10003" --proxy-user "YOUR_USERNAME" "https://chatgpt.com/"

-o NUL discards the response body so the terminal isn’t flooded with an entire web page. The connection trace still appears.

The important part of the failed record was:

1
2
3
4
5
6

> CONNECT chatgpt.com:443 HTTP/1.1
> Host: chatgpt.com:443
> Proxy-Authorization: Basic [REDACTED]
...
* Recv failure: Connection was reset
curl: (56) Recv failure: Connection was reset

Some later tests instead showed:

1

curl: (56) Proxy CONNECT aborted

Only then did I begin to understand that accessing an HTTPS website through an HTTP proxy isn’t simply a matter of throwing the web request at it.

First, the client tells the proxy: please establish a tunnel to chatgpt.com:443. That is CONNECT. After receiving a successful response, the client performs a TLS handshake with the target website inside that tunnel, then sends the encrypted web request.

So 200 Connection established followed by 200 OK isn’t the same success message repeated twice. The first means the proxy tunnel was established. The second is the target website’s HTTP response.

My failed trace hadn’t even received the first one.

That showed the connection was interrupted during CONNECT, before the TLS handshake with the target website. Whether the request was interfered with along the way or something went wrong in proxy-side processing, the client log didn’t say. Locating the stage of interruption doesn’t identify the party responsible.

Likewise, Proxy auth using Basic only says the client was preparing to send that form of authentication. It doesn’t mean the server had confirmed authentication was successful.

An email from the technical team sent me in another direction

Support passed the ticket to the technical team. Initially, they said a reply would usually come within two business days, but I received an email later that same day.

They noticed the same contrast: their checking endpoint was reachable, while Google and ChatGPT were reset. They thought it looked like network-level filtering or interference, asked where I was located, and suggested comparing protocols and results with the VPN both on and off.

The technical team’s reply.

So I switched OuO back on and enabled the client’s TUN mode, which uses a virtual network interface.

I learned another lesson here: TUN isn’t an overseas node or an encryption protocol. Its job is to hand system network packets to the proxy program. Whether they are encrypted afterward and which exit they take depends on the client and the actual node protocol.

With only the system proxy enabled, applications have to use those settings to send requests into it. Since I’d explicitly specified Decodo in curl, I couldn’t expect it to automatically add OuO as another layer. I used TUN for this comparison so that the connection to Decodo’s entry point would also enter the existing proxy path.

This time, Google worked, and YouTube was accessible too.

But ChatGPT still returned:

1

Proxy CONNECT aborted

I also tested openai.com and api.openai.com. The same error.

It was starting to look as though OpenAI was restricted separately. Yet support had explicitly said it was supported.

There was still one variable from the email I hadn’t finished trying.

SOCKS5.

I switched to SOCKS5, and hundreds of KB of web page suddenly filled the terminal

SOCKS5 is another proxy protocol. Instead of negotiating through HTTP’s CONNECT method, it has its own authentication and connection-request formats, allowing the proxy to connect to a specified domain and port.

I switched to this command:

1

curl.exe -v -o NUL --proxy "socks5h://isp.decodo.com:10003" --proxy-user "YOUR_USERNAME" "https://chatgpt.com/"

The h in socks5h matters: it tells the proxy to resolve the target domain, rather than having the local machine resolve it to an IP first.

On the very first attempt, I hadn’t added -o NUL yet. As soon as I pressed Enter, a huge amount of text streamed through the terminal—far more than I could copy.

I was still asking how I was supposed to capture all of that.

Then I realized it wasn’t a long stream of errors.

It was the body of ChatGPT’s web page.

After adding -o NUL, the key result was easy to see:

1
2
3
4
5
6
7

* SOCKS5 connect to chatgpt.com:443 (remotely resolved)
* SOCKS5 request granted.
...
> GET / HTTP/1.1
> Host: chatgpt.com
...
< HTTP/1.1 200 OK

Then I turned off both OuO’s system proxy and TUN and connected directly from my current network to Decodo SOCKS5.

Still 200 OK. This attempt downloaded about 486 KB of page content.

After all that tinkering, I finally had something that worked.

The HTTP CONNECT and SOCKS5 test results.

Diagram guide: the same entry point, two negotiation methods. HTTP sent CONNECT but received no tunnel-success response, recording a reset / CONNECT abort before TLS to the target. SOCKS5 negotiated authentication and the target domain, received SOCKS5 request granted, then reached HTTP 200 OK after TLS. This describes this test, not a general protocol ranking or a verified cause; SOCKS5 is not inherently encrypted, and one page response does not verify every feature.

HTTP proxies, HTTPS proxies, and SOCKS5: what’s the difference?

Changing protocols solved this connectivity problem, but the distinction between them isn’t a question of one being more advanced.

When accessing an HTTPS site through a plain HTTP proxy, TLS can still encrypt the target content, but that doesn’t automatically encrypt the initial CONNECT negotiation itself. With a true HTTPS proxy, the client first establishes TLS to the proxy as well. A proxy doesn’t become an HTTPS proxy merely because the target URL starts with https://.

And SOCKS5 isn’t inherently an encrypted tunnel either. In particular, the standard explicitly states that the password in the common username/password authentication method is carried in plaintext. The target website’s HTTPS protects web content; it doesn’t reach backward and encrypt the SOCKS5 authentication stage.

What this test confirmed was that, with the same set of entry points, HTTP CONNECT failed to access OpenAI while the SOCKS5 path worked. Whether the difference came from protocol characteristics, gateway implementation, or intermediate network equipment remained unresolved.

Being able to connect and being able to get work done are different things

I thought changing every type: http to type: socks5 in FlClash would finish the job.

Instead, the nodes alternated between green and timeout. The websites weren’t completely inaccessible: they opened, but slowly, with resources loading endlessly and spinners wearing out my patience.

That was even more annoying than a page that wouldn’t open at all, because you always felt it might work if you waited just a little longer.

A successful curl request proves that that particular request received a response. A browser still needs to load scripts, images, and API requests. Some ChatGPT and Codex features also use WebSockets—connections supporting ongoing two-way communication.

Downloading the homepage HTML was still some distance from a useful setup.

I made another comparison. In both cases, I requested the ChatGPT homepage through Decodo SOCKS5. In one, OuO TUN took over the connection first; in the other, I connected directly to Decodo.

These are the raw numbers I saved:

Time to first byte measures from the start of the request until the first byte arrives, including connection setup and server processing. Total time is the duration of the whole operation. The rate here is also the average transfer rate of this response, not a separate measurement of line bandwidth.

One request over each path: direct access and access through OuO.

Chart guide: left is direct to Decodo, right is OuO TUN → Decodo. Rows show time to first byte, total request time, and average response transfer rate. This extra-hop path was faster on this request. Body size, cache state, and test timing were not controlled, so the figure is not a performance benchmark or a long-term speed guarantee. It omits a cross-path time_connect comparison because TUN can change what that measurement means.

Each path was recorded only once. Response-body sizes and caching weren’t controlled, so these numbers can’t be used directly to calculate a long-term percentage speedup.

In particular, time_connect of just 9 milliseconds on the TUN side doesn’t mean the actual latency to the overseas proxy was only 9 milliseconds. Once a virtual network takes over, the connection setup curl sees does not necessarily mean the entire cross-border path behind it is ready.

Those numbers made me more willing to try chaining. What ultimately convinced me to keep it, though, was the later experience of actually using the browser and Codex.

I also stopped planning to replace the old proxy directly with the new one.

Instead: let the old proxy provide a smoother route, and let the new proxy determine the final exit IP.

In the end, I chained the old and new proxies together

I had also worried about a very practical issue: I often watch courses on YouTube while asking AI questions. Would I now have to use OuO for video, then switch to Decodo to ask a question?

I’d spend all my study time switching nodes.

A proxy client can route by domain, so there is no need to keep switching. Different connections in the same browser on the same device don’t have to use the same exit.

I eventually split the traffic into three paths:

1
2
3
4
5
6
7
8

ChatGPT / OpenAI
设备 → OuO 日本节点 → Decodo SOCKS5 → OpenAI

Google / YouTube / Gmail 等普通海外访问
设备 → OuO 香港节点 → 目标服务

命中原有直连规则的国内及本地服务
设备 → 直接连接

Route guide: ChatGPT / OpenAI uses device → OuO Japan node → Decodo SOCKS5 → OpenAI. Ordinary overseas access, including Google / YouTube / Gmail, uses device → OuO Hong Kong node → target service. Domestic Chinese and local services matching existing direct rules connect directly.

The final traffic routing and proxy chain.

Diagram guide: separate “how to reach it” from “which IP to exit through.” Clash or Shadowrocket applies rules on the computer / iPhone. Purple is OpenAI traffic via OuO Japan, then the fixed dedicated Japanese Decodo exit; green is Google / YouTube and similar traffic via OuO Hong Kong; orange is traffic matching DIRECT rules. Arrows show actual connection direction. Policy groups are routing logic, not extra servers; TUN is packet interception, not an additional node or encryption protocol.

In this chain, OuO’s Japan node was the first hop, and Decodo was the exit that sent the final request outward. For connections that actually matched this chain, the target service saw Decodo’s exit, rather than OuO’s shared exit.

Proxy chaining separates routing and exit responsibilities; it isn’t a guarantee of anonymity. OuO and Decodo still each participate in the connection. SOCKS5 authentication and target information also don’t automatically become confidential from every participant along the entire chain.

Windows: a nineteen-thousand-line configuration, with only a few key changes

My OuO configuration had more than nineteen thousand lines.

Looking at it, I wasn’t very keen to edit it by hand. I eventually gave the file to AI to help inspect and modify.

Fortunately, the existing configuration already had policy groups such as OpenAI, Google, and YouTube. A policy group lets a category of requests select a node—or select another group.

There was no need to rewrite all the later rules. The key was to add dialer-proxy to the Decodo nodes, referring to the existing Japan node group so Decodo would establish its connections through that group.

Within the existing YAML configuration’s proxies list, these were the key fields for a Decodo node:

1
2
3
4
5
6
7
8

proxies:
  - name: "Decodo JP Main"
    type: socks5
    server: isp.decodo.com
    port: 10003
    username: "YOUR_USERNAME"
    password: "YOUR_PASSWORD"
    dialer-proxy: "🇯🇵 日本节点"

Then I pointed the OpenAI policy group at the Decodo selection group and selected Decodo JP Main. The first-hop Japan group contained only the original OuO nodes. I didn’t put Decodo back into it, which would risk a loop.

This chaining configuration is separate from TUN. Earlier, TUN was part of the experiment to intercept curl’s connection. Once the configurations were formally merged, dialer-proxy explicitly specified the first hop. With only system proxy enabled and TUN off, my computer could still put requests entering the client through this chain. That did not take over every program’s traffic.

The three Decodo nodes in FlClash.

Screenshot guide: the selected group is “Decodo fixed IP,” with Decodo JP Main selected. The neighboring tab labeled “manual switch” is another policy group; the button runs latency tests.

Once the websites worked again, I checked the actual exit once more.

I temporarily assigned ip.decodo.com to the OpenAI policy group too, then queried the final exit and confirmed that it really was the primary Japanese IP. Without that rule, an ordinary IP-check page would use the default overseas exit, so seeing an OuO address there was normal.

Then I sent a small task in Codex that actually called the model and went back to the client’s connections page.

Codex matched Decodo JP Main.

Screenshot guide: the connection-details labels are creation time, process, network type, rule, host, source, upload, download, DNS mode, remote target, and proxy chain. The process is codex.exe, the host is chatgpt.com, and the chain includes Decodo JP Main; the entry address is hidden.

The process shown was codex.exe, the target was chatgpt.com, and the policy chain included Decodo JP Main. This model request really had reached the node I’d specified.

The second computer taught me another lesson: a policy group isn’t a global master switch

I have two desktops and a laptop, and the clients aren’t all the same: some use FlClash, others Clash Verge.

After the first machine worked, I copied the configuration over. ChatGPT worked and YouTube played, but Gmail stayed on its loading screen.

I thought the route had broken again on the new computer.

It turned out that this machine’s Google group was still set to manual switching, which pointed to another OuO route. Once I changed it to a working Japanese node, Gmail recovered. Later, based on actual use, I chose a low-latency Hong Kong node for ordinary overseas traffic.

I’d assumed that once I’d selected Decodo JP Main, I no longer needed to worry about the other groups below it.

In rule mode, though, group choices can be active at the same time. Google follows the Google group, YouTube follows the YouTube group, and OpenAI follows the OpenAI group. One group can also refer to another, so they aren’t entirely independent either.

Copying the YAML doesn’t mean every client has exactly the same saved selection state.

Once I understood that, moving the setup to the remaining computers went smoothly.

I kept a local copy of the configuration in the end, so subscription updates wouldn’t overwrite my manual changes. But that copy won’t automatically track OuO updates either. If the subscription’s nodes change later, I’ll need to sync them and check the references.

The final hurdle: moving the same logic to the iPhone

My phone uses Shadowrocket, so I couldn’t expect to copy Mihomo’s fields over directly.

But in the node details I found an option that was very easy to understand: Proxy Through.

I filled in the SOCKS5 parameters for Decodo JP Main and set Proxy Through to OuO’s AnyTLS.JP 03. That gave me the corresponding network path.

Shadowrocket’s Proxy Through setting.

Screenshot guide: the fields are type, address, port, user, password, algorithm, plugin, TCP Fast Open, UDP relay, Proxy Through, and remarks. The node uses SOCKS5 at isp.decodo.com:10003 and passes through AnyTLS.JP 03; account and password are hidden.

I first temporarily sent the phone through Decodo, opened the checking page in Safari, and got the same exit as on the computer. The ChatGPT web version opened too.

But I wasn’t sure whether the app was using it as well.

So I went back to the logs.

In the earlier round, before fine-grained routing, ios.chat.openai.com and chatgpt.com used PROXY, Google used PROXY too, and the currently selected node on the home screen was Decodo.

That meant Google also went through Decodo. Routing wasn’t finished yet.

I put the OpenAI-related rules at the beginning of the current configuration, explicitly pointing them to Decodo JP Main. Then I changed the home screen’s default node back to OuO Hong Kong and kept Global Routing in Configuration mode.

These were the four core rules. My actual configuration also added DOMAIN-KEYWORD,openai, pointing to Decodo as well, to match connections whose domain contains openai.

1
2
3
4

DOMAIN-SUFFIX,chatgpt.com,Decodo JP Main
DOMAIN-SUFFIX,openai.com,Decodo JP Main
DOMAIN-SUFFIX,oaistatic.com,Decodo JP Main
DOMAIN-SUFFIX,oaiusercontent.com,Decodo JP Main

Suffix matching covers the corresponding subdomains, so I didn’t need separate rules for ios.chat.openai.com, ws.chatgpt.com, and ab.chatgpt.com.

After the changes, the key log entries looked like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# ChatGPT App
url = ios.chat.openai.com:443
result = DOMAIN-KEYWORD,openai,DECODO JP MAIN # Decodo JP Main

# ChatGPT 的其他连接
url = ws.chatgpt.com:443
result = DOMAIN-SUFFIX,chatgpt.com,DECODO JP MAIN # Decodo JP Main

# 用户内容域名
url = sdmntprcentralus.oaiusercontent.com:443
result = DOMAIN-SUFFIX,oaiusercontent.com,DECODO JP MAIN # Decodo JP Main

# Google
url = www.google.com:443
result = DOMAIN-KEYWORD,google,PROXY # 🇭🇰 AnyTLS.HK 03

# YouTube
url = www.youtube.com:443
result = DOMAIN-KEYWORD,youtube,PROXY # 🇭🇰 AnyTLS.HK 03

# 国内网站
url = m.baidu.com:443
result = DOMAIN-SUFFIX,baidu.com,DIRECT

Log guide: the Chinese comments identify other ChatGPT connections, the user-content domain, and a domestic Chinese website, respectively. The original log text is retained.

The same batch of records also showed Decodo’s backend chain, the OuO Japan session, and the SOCKS5 handshake ok.

The app’s main connections went through Decodo; Google and YouTube went through Hong Kong; domestic websites stayed direct. Routing matched what I’d intended. Along with actually sending messages and loading content, that established that everyday phone use worked too. I haven’t fully tested voice and video features.

Only at this point did the three computers and iPhone finally share the same main routing logic.

After all that, what had I actually solved?

At the start, my thinking was: I wanted a clean IP, preferably one that would stop me worrying about GPT getting dumber.

What I actually solved was more specific: my main OpenAI requests now had a path with a fixed exit, inspectable routing, and smooth performance in actual use.

Exit reputation, access protocol, connection quality, and routing rules need separate checks. A low risk score doesn’t guarantee a target will be accessible. A 200 from curl doesn’t guarantee a smooth browser experience either. Only when actual requests, client logs, and everyday use lined up did I consider the configuration working.

AI saved me a lot of trouble by explaining concepts, organizing commands, and editing a configuration with more than ten thousand lines. But it also gave PowerShell a CMD command and guessed at causes too early from incomplete symptoms. The configuration I kept was the result of repeated real tests.

To finish up, I used find and replace to update the passwords on the three Decodo nodes, then synced them to the other devices. The Proxy-Authorization: Basic ... entries in troubleshooting logs also needed redacting: Base64 is encoding, not a way to keep a secret.

As for the original question: did giving GPT a clean IP actually make it smarter?

I didn’t run a before-and-after comparison of model capability, so this experience can’t answer that.

What I can confirm is that ChatGPT is stable in use and Codex’s model connection shows the right policy chain, while YouTube and Gmail can keep using the original, faster routes.

Maybe my account had never been “dumbed down” in the first place.

I’ve recently been taking a course on Agentic AI. One lecture, Post-Training Verifiable Agents, gave me a basic understanding of concepts such as agent, tool, harness, verifier, and reward, along with an introductory grasp of methods including SFT, RL, Best-of-N, and Rejection Sampling. But I hadn’t actually run the whole system myself.

So I decided to do a Lab to reinforce what I’d learned. There were two main things I wanted to understand: how much difference there would be between One-shot inference and an Agent Loop using the same LLM, and how much Test-time Sampling would improve the Agent Loop as I added more rollouts. I used SWE-bench Lite as the benchmark.

1. Pilot: preparing the experiment

1.1 Experimental framework and design

The experiment called the Deepseek-V4-Flash Thinking model.

A was the One-shot baseline. The model received an issue and roughly 13K tokens of static repository context retrieved with BM25. It made a single API request, used no tools, received no environmental feedback, and directly output a Patch. This measured the same model’s ability to produce a repair in one go using only fixed context.

B was an Agent Loop with access to list_tree, read_file, search_code, edit_file, write_file, run_shell, and finish. The harness returned each execution result to the model as an Observation. The model could then continue reasoning, editing, or testing until it called finish itself or reached the step limit.

Each SWE-bench Lite task provided a real issue and the corresponding repository’s Base Commit. A and B both ultimately produced a Candidate Patch, which the official verifier evaluated in an isolated evaluation environment to determine whether the task was solved. B’s Agent Loop itself ran in a separate Docker container.

Diagram guide: SWE-bench Lite tasks branch into A · One-shot → Patch → verifier → reward, and B · Agent Loop → trajectory → Patch → verifier → reward. Eight B rollouts feed C · Pass@k; reward=1 trajectories feed D · Rejection Sampling and then a demonstration dataset.

A and B were parallel method comparisons. C evaluated repeated B rollouts, while D processed trajectories that had already finished. The four parts shared tasks and artifacts; a single trajectory did not pass sequentially from A through D.

C ran 8 independent B rollouts for each task and calculated Pass@1, Pass@2, Pass@4, and Pass@8. Rollouts shared neither conversations nor workspaces. This measured how many tasks the candidate set could cover as sampling increased.

D was offline Rejection Sampling. It retained B trajectories with reward=1, rejected those with reward=0, and then deduplicated Patches within each task to produce data that could be used later.

1.2 Agent Loop and Test-time Sampling

The Agent Loop ran in a separate container without network access or mounted host directories. Every rollout started from the same Base Commit. Its workspace did not inherit the previous Patch, and it could not see other rollouts’ trajectories. The official evaluator ran only after the agent finished. This kept the Gold Patch, earlier sampling results, and evaluation details separate. The Gold Patch was used to confirm that the benchmark and evaluation environment worked; it can be understood as the official answer, and was never included in the prompt, workspace, or information returned by tools.

Diagram guide: LLM → tool call → execution in the environment → Observation → further reasoning. At finish or the step limit, the workspace yields a Patch for the verifier, which returns a reward. Local tests feed Observations, not the final reward.

Passing local tests through run_shell did not mean a task had earned reward=1. Local tests might provide incomplete coverage, and later edits could break behavior that had previously passed. The experiment recognized only the verifier result for the final Candidate Patch.

1.3 Pilot v1

Pilot v1 started with four tasks and returned A=0/4 and B=1/4. After checking the trajectories and logs, I found that these two numbers weren’t directly comparable.

At the time, A had max_tokens set to 8192, with reasoning and the final answer sharing that output budget. Of the four One-shot runs, 3/4 hit the token limit, and 4/4 final Patches were empty. The token limit was too low.

B’s problem was more straightforward: all 15 calls to apply_patch failed, and 4/4 B trajectories hit the 30-step limit. Even when the model found the right location, it had no reliable editing interface.

With all of A’s Patches empty and B’s editing interface broken, there was no point continuing to compare 0/4 with 1/4. I moved straight on to Pilot v2.

1.4 Pilot v2: fixing the harness and validating the protocol

Pilot v2 first addressed the problems exposed by Pilot v1. I raised One-shot’s max_tokens from 8192 to 65536. B stopped using apply_patch and switched to edit_file and write_file; its step limit also rose from 30 to 60. The goal of this round was to confirm that A and B could run properly, not to chase a higher score.

After repairing the interfaces, I checked B’s diversity. I ran 4 independent B rollouts on the same Astroid task. They took 39, 26, 40, and 34 steps and produced 4 different Patch hashes. They used the same Base Commit, prompt, model, and harness configuration, with isolated containers. This was only a diversity check, not a Pass@k calculation. It confirmed that the same model could produce different trajectories on the same task.

All four One-shot runs in Pilot v2 produced nonempty Patches, with no further output truncation. All four main B trajectories called finish themselves, none reached the 60-step limit, and edit_file produced no errors. At this point, the harness was basically stable and ready for the Formal phase.

1.5 Freezing the experimental protocol

Formal used 20 tasks from 10 repositories, with two tasks per repository. A ran once per task and B ran 8 times, producing 20 A trajectories and 160 B trajectories: 180 formal trajectories in total.

Diagram guide: Pilot v1—8192 tokens, failed apply_patch, 30-step limit, measurement setup debugging. Pilot v2—65536 tokens, edit_file / write_file, 60-step limit, protocol validation and diversity checks. Formal—20 tasks, 10 repositories, A×1 / B×8, 180 trajectories, frozen protocol.

Before entering Formal, I froze the model, prompt, tools, max_steps, token budgets, verifier, reward classification rules, and task set. Changing the prompt, adding steps, or supplying more tools whenever a task failed would turn formal evaluation into continuous tuning against the benchmark. So even when a task scored 0/8, or a trajectory remained unfinished at step 60, I retained the original result and left changes for the next experiment.

One detail worth mentioning: at 20/180, Formal was interrupted by a failure to write a checkpoint on Windows. I made a fix limited to saving checkpoint state. I did not change the model, prompt, tools, steps, verifier, or reward, and did not rerun completed trajectories. After the fix, the same Formal run continued with protocol_changed=false.

2. Results

2.1 One-shot versus Agent Loop: 20% vs 55%

The Formal results were clear: One-shot solved 4/20 tasks, or 20%. Of the Agent Loop’s 160 rollouts, 88 succeeded, giving a task-averaged Pass@1 of 55%. The difference was 35 percentage points.

Chart guide: the vertical axis is resolution rate (%); gray is One-shot and red is Agent Loop. The bracket marks a 35-percentage-point difference for the same model.

One-shot saw only static repository context. The Agent Loop could inspect code, edit, test, and adjust its next actions using Observations. The 35-percentage-point difference came from this overall change in interaction, and cannot be attributed to an individual tool or a particular piece of test feedback. The comparison is also limited to this frozen set of 20 tasks.

Each task had 8 B rollouts, for 160 across 20 tasks, of which 88 had reward=1. Averaging c/8 across tasks also gives 88/160=55%. This happens to equal the success proportion across all trajectories, but Pass@1 is still defined by task. The subsequent Pass@2, Pass@4, and Pass@8 use the same calculation.

2.2 Pass@k

Formal sampled a fixed 8 rollouts per task. Let n=8, c be the number of successful rollouts, and k be the number of candidates drawn from the observed samples. When k=n=8, a task’s Pass@8 is 1 whenever c>0.

With more independent B rollouts, Pass@1, Pass@2, Pass@4, and Pass@8 were 55.00%, 61.7857%, 64.6429%, and 65.00%, respectively. Increasing from 1 candidate to 2 improved coverage by 6.79 percentage points; going from 2 to 4 added 2.86 points; going from 4 to 8 added only 0.36 points.

Specifically, when k=n=8, a task’s Pass@8 is 1 as long as c>0. In this experiment, 13/20 tasks succeeded at least once, so both average Pass@8 and Solved@8 were 65%.

Chart guide: the horizontal axis is candidate/rollout count k; the vertical axis is Pass@k (%). Red annotations show gains in percentage points. The dashed gray line is One-shot at 20%.

Pass@k, Best-of-N, and Rejection Sampling serve different purposes. Pass@k asks whether a successful result exists among the candidates. Best-of-N additionally needs a selector to choose one of N candidates. Rejection Sampling uses a verifier to filter trajectories for data generation. This experiment had no selector, so Pass@8 cannot be treated directly as a deployment success rate.

2.3 Task-level c/8 and the gains from sampling

Breaking the aggregate curve down by task, 7 of the 20 tasks scored 0/8, another 7 scored 8/8, and only 6 fell between those extremes. Those 6 succeeded 3, 5, 5, 6, 6, and 7 times, respectively. No task scored 1/8 or 2/8.

Chart guide: each row is a task, ordered by successful rollout count on the horizontal axis. Orange marks 0/8, green marks 8/8, and red marks intermediate results. Each task has 8 independent rollouts.

This also explains why Pass@k saturated quickly. More samples could not increase coverage for the 7 tasks already at 8/8. The 7 tasks at 0/8 had no successes within the current 8 samples, so the additional gains mainly came from the 6 tasks in between. Here, 0/8 describes only what was observed in this experiment; it does not mean that further rollouts would necessarily be useless.

2.4 Steps and costs for successful and failed trajectories

Of the 160 B trajectories, 88 succeeded and 72 failed. Successful trajectories used a mean of 35.26 steps, with a median of 32. Failed trajectories used a mean of 45.5 steps, with a median of 52.5.

Chart guide: green means success and orange means failure. The horizontal axis is step count; the vertical axis is cumulative proportion (%). Dashed lines mark the two medians.

The ECDF (empirical cumulative distribution function) above includes every trajectory, with no smoothing or removal of extreme points. At any step position, it shows the proportion of trajectories that had ended by that point. The failure curve shifts right across the distribution as a whole; the difference is not just a few 60-step samples pulling up the mean.

Successful trajectories had a mean API cost of 0.06984 CNY and a median of 0.05082 CNY. Failed trajectories had a mean cost of 0.13955 CNY and a median of 0.11717 CNY. The failed group’s mean cost was about twice the successful group’s, and its median was more than twice as high. Costs across all samples ranged from 0.01028 to 0.39885 CNY, and the ECDF retains the high-cost tail.

Chart guide: green means success and orange means failure. The horizontal axis is cost (CNY); the vertical axis is cumulative proportion (%). The dashed median labels are rounded to 0.051 and 0.117 CNY.

Chart guide: the horizontal axis is step count and the vertical axis is cost (CNY). Green circles are successful trajectories; orange triangles are failed trajectories. All 160 formal B trajectories are included.

The scatter plot fills in the relationship between steps and cost. There are more failed points in the high-step region, but costs at the same step count still vary with input tokens, output tokens, and cache hits. This is a correlation; it does not establish that longer execution causes failure. Steps describe interaction length, while CNY records actual API consumption.

2.5 Differences in tool use

In average calls per trajectory, the failed group used read_file 15.03 times versus 9.64 for the successful group; search_code was 10.15 versus 6.09, edit_file was 4.43 versus 2.93, and run_shell was 22.92 versus 19.40. Failed trajectories also used list_tree more, while successful ones used write_file and finish slightly more.

Chart guide: the horizontal axis is average tool calls per trajectory. Green circles mean success and orange triangles mean failure; each row names a tool.

Failed trajectories tended to spend more calls searching, editing, and testing. Some of that was necessary investigation, and some was repeated reading and repeated Patch revisions. The fact that write_file and finish did not rise alongside the other counts also reminded me not to look only at totals. finish means the agent chose to stop; a read_file call might retrieve a crucial fact or merely reread the same code. Call order, arguments, and the Observations returned afterward tell us more than counts alone.

2.6 Step limits and outcomes

In Formal B, 46 trajectories reached the 60-step limit: 15 succeeded and 31 failed, a success rate of 32.61%. Of the 114 that did not reach it, 73 succeeded and 41 failed, a success rate of 64.04%.

Chart guide: the left bar reached the step limit; the right did not. Green is success and orange is failure. The vertical axis shows percentage composition; numbers inside the bars are trajectory counts.

There were still 15 successes among the 46 that reached the limit, and 41 failures among the 114 that did not. The step limit was associated with outcome, but it could not replace an examination of what happened within a trajectory.

Failed trajectories ran longer and cost more, and 46 trajectories hit the limit. This made me wonder whether 60 steps was too restrictive. The aggregates couldn’t tell me whether those trajectories were still making progress at the boundary or had already fallen into repeated searches, so I went on to inspect the 7 tasks with c=0.

3. Failure analysis of c=0 tasks

3.1 Classifying 56 failed trajectories

The 7 c=0 tasks each had 8 rollouts without a single success, giving 56 trajectories with reward=0. After Formal finished, I analyzed them individually. I mainly looked at whether the final Patch touched relevant code, whether there were new edits or new test feedback late in the run, and whether the trajectory showed repeated searching, back-and-forth Patch changes, or scope drift. This subset consists of tasks that remained unsolved across all 8 samples; it does not represent all 72 failed trajectories.

The classification was a retrospective, heuristic, qualitative judgment, not a ground-truth label supplied by the verifier. The analysis also did not randomly vary max_steps or replanning strategies. I classified the 56 trajectories as follows: 35 capability/semantic failures (likely_capability_failure), or 62.5%; 11 strategy failures (likely_strategy_failure), or 19.6%; 7 horizon-limited failures (likely_horizon_limited), or 12.5%; and 3 mixed/unclear cases (mixed_or_unclear), or 5.4%.

Chart guide: the horizontal axis counts trajectories. The four bars are capability/semantic, strategy, horizon-limited, and mixed/unclear failures; labels show counts and percentages from the offline classification.

Of the 56 trajectories, 22 hit the step limit, but only 7 were classified as horizon-limited. That classification additionally required edits near the end to remain localized, a nonempty Patch relevant to the issue, and tests that still produced new or improving feedback, without stagnation signals already dominating the run. Hitting the step limit alone does not establish that the failure was horizon-limited.

The capability/semantic category describes cases where the agent found relevant code and made relevant edits but did not satisfy the full software behavior constraints. It characterizes the current trajectory, not a ceiling on the model’s ability.

3.2 Three representative tasks

All seven tasks scored 0/8, but their failure processes differed. All 8 trajectories for astropy-7746 and flask-4992 were capability/semantic failures. xarray-3364 had 3 strategy failures, 3 horizon-limited failures, and 2 mixed cases. The remaining tasks had other combinations.

Chart guide: the horizontal axis counts failed rollouts. Blue is capability/semantic, ochre is strategy, purple is horizon-limited, and gray is mixed/unclear.

The 8 flask-4992 rollouts quickly found Config.from_file, the relevant documentation, and tests. All called finish themselves, and none hit the 60-step limit. They tended to accept complete file-opening modes such as r and rb, while the actual call in the issue used mode='b'. The edit location was right, and some local tests passed. What they missed were the composition rules for file-opening modes and the public API semantics. The agents finished in 17 to 23 steps; raising the limit would not address this kind of omission.

All 8 xarray-3364 rollouts used the full 60 steps. Edits concentrated on concat.py, repeatedly dealing with variable discovery, the choice between merging and concatenation, dimensions, dtype promotion, and fill-value construction. Three were still making localized edits and receiving new or improving feedback at the end, so I classified them as horizon-limited. Another 3 were strategy failures and 2 were mixed. In the remaining processes, I saw repeated searching, rereading, and testing; some were still examining installed-package caches and external references late in the run. The final Patches remained incomplete or internally inconsistent. All 8/8 reaching the limit does not mean all 8/8 lacked enough execution steps.

All 8 astropy-7746 rollouts located the WCS array conversion and added an early return for empty input. Most trajectories received passing local-test feedback before calling finish themselves, and none hit the step limit. The final omissions involved constraints on output containers, shape semantics, axis counts, and ra_dec_order. They found the right code and passed local tests, but the full behavior still failed the verifier.

3.3 From adding steps to replanning

Before the failure analysis, I saw that failed trajectories ran longer on average and frequently hit the step limit, so I initially thought 60 steps wasn’t enough. When I examined the 7 c=0 tasks, though, 22 trajectories had hit the limit but only 7 looked more like horizon-limited cases. I began to realize that many failures were stuck on semantics, consistency across code paths, or search strategy—problems that simply running longer wouldn’t solve.

A longer execution budget might help a few trajectories. The 3 horizon-limited samples from xarray-3364 would be worth testing separately. But if every run were uniformly increased from 60 to 100 steps, trajectories already repeating searches, producing no new evidence, or drifting in scope would keep consuming budget too. Whether reward would improve remains unknown.

If I do another experiment, I’d try adaptive computation: selectively extend the budget when late edits remain focused and test feedback is still changing; trigger replanning or early stopping when searching and editing start repeating. Keeping the model, prompt, and tools unchanged and changing only the conditions for assigning extra steps would make the source of any change clearer.

3.4 Rejection Sampling

Among the 160 formal B trajectories, the verifier assigned reward=1 to 88 and reward=0 to 72. Rejection Sampling rejected the 72 failures, retained the 88 successes, and deduplicated Patches within each task to obtain 82 demonstration examples.

Diagram guide: the verifier splits 160 B trajectories into 88 with reward=1 and 72 with reward=0. The latter are rejected; the former pass through Patch deduplication into 82 demonstration records. No SFT is performed in this Lab.

Filtering used only verifier results, with no manual selection. At this point, the Generate → Verify → Filter data-generation pipeline was working. I did not proceed to SFT: the harness, Agent Loop, Test-time Sampling, failure analysis, and Rejection Sampling this Lab set out to examine already had actual results. Adding a training stage would not answer the original three questions.

4. Looking back

The Pilot phase took a fair amount of time and tokens, but that expense was necessary. Pilot first eliminated harness problems involving output budgets, editing interfaces, and step limits, so Formal wouldn’t end up measuring faults in the experimental apparatus itself.

Formal’s most direct finding was that the Agent Loop substantially outperformed One-shot. With the same model, the same 20 SWE-bench Lite tasks, and a frozen experimental protocol, One-shot’s resolution rate was 20%, while Agent Loop Pass@1 was 55%. In other words, replacing static one-off inference with an Agent Loop that could actively retrieve code, use tools, run tests, and keep adjusting based on Observations improved single-run resolution by 35 percentage points. This was the clearest gain in the experiment.

The second finding was that Test-time Sampling helped, but its returns diminished quickly. The increase from 55% at Pass@1 to 65% at Pass@8 shows that running more independent rollouts did cover some single-run instability. But going from 4 rollouts to 8 added only 0.36 percentage points. Task-level results help explain this: 7 of the 20 tasks scored 8/8 and 7 scored 0/8, so most additional sampling gains came from the 6 in between. For the 0/8 tasks, I can only say that no success was observed in these 8 samples. I cannot conclude that further rollouts would necessarily be useless.

The third finding was that more computation did not automatically produce better results in this set of experiments. Failed trajectories ran longer on average and cost more. Of the 56 c=0 failures, 22 hit the step limit, but only 7 looked more like cases limited simply by execution length. Many trajectories had already found the relevant code and even passed some local tests, yet still missed API behavior, edge cases, output semantics, or consistency between code paths. At least on these results, uniformly raising max_steps above 60 does not look like the most promising first change.

So if I continue with another experiment, I’d optimize how computation is allocated: give more steps to trajectories still producing new feedback, trigger replanning when searches or edits start repeating, and stop early when needed. More than simply increasing rollouts or steps, what I now want to understand is: when is it worth continuing to compute, and when should I change direction?

CCPC 2021 Online F - Nun Heh Heh Aaaaaaaaaaa

We are given several strings containing only lowercase English letters. For each string, count its subsequences of the form nunhehheh followed by at least one a:

1

nunhehheh + 至少一个 a

A subsequence only requires the chosen indices to be strictly increasing; the characters need not be contiguous. Even if the resulting strings are identical, different selections of indices count separately. The answer is taken modulo $998244353$.

The number of test cases satisfies $T\le 1000$. Each string has length at most $10^5$, and the total length of all strings is at most $10^6$.

This is a fixed-pattern subsequence-counting DP problem. The code itself is short. What is worth reviewing is how to partition the choices, and how to distinguish cumulative states from states newly formed at the current position.

Partitioning by the Last h

A valid subsequence consists of two parts:

1

nunhehheh

followed by a nonempty sequence of as.

Suppose the final h of nunhehheh is chosen at position $i$, with $c_i$ occurrences of a after $i$. Each suffix a may be chosen or skipped, giving $2^{c_i}$ subsets. Excluding the empty choice leaves:

Let $f_i$ be the number of ways to choose nunhehhe before $i$. Fixing $i$ as the final h gives this many complete choices:

The total answer can therefore be written as:

This partition causes no double counting. Every valid subsequence has a unique final h in its nunhehheh, so it belongs to exactly one position $i$. Conversely, combining a choice of the first eight characters, the fixed ninth character h, and a nonempty choice of suffix as always produces a valid subsequence.

Maintaining Only the First Eight Characters

The complete fixed prefix is nunhehheh, but its last h already partitions the answer. The DP therefore only needs to maintain the preceding eight characters:

1

nunhehhe

Call this length-$8$ pattern $P$. While scanning a prefix of the original string, define $D_j$ as the number of ways to choose the first $j$ characters of $P$ from the characters scanned so far:

Here, $0\le j\le 8$. In particular, $D_0=1$, since there is exactly one way to choose the empty string: choose nothing. Initialize every other state to $0$. The implementation stores $D_j$ in dp[j].

When scanning character $s_i$, if it equals the $j$th character of $P$, append it to every choice that has already matched the first $j-1$ characters:

The corresponding one-dimensional transition is:

1
2
3
4
5

for (int j = 8; j >= 1; --j) {
    if (s[i] == temp[j - 1]) {
        dp[j] = (dp[j] + dp[j - 1]) % mod;
    }
}

We must update in descending order. Suppose the pattern were nn. With an ascending update, scanning one n would first update $D_1$, then immediately use that newly updated $D_1$ to update $D_2$, effectively selecting the same original position twice. Descending updates ensure that the state on the right-hand side still comes from before the current character was added, so each position is used at most once.

When the current character is h, $D_8$ before the update is exactly the $f_i$ defined above. Semantically, we can add this position’s contribution to the answer before performing the DP transitions. The accepted code writes them in the opposite order, but the current h cannot update $D_8$, which ends in e, so the two orders are equivalent here.

Nonempty Choices of Suffix a Characters

To obtain the number of as to the right of each position, precompute suffix counts from right to left:

The implementation stores these counts in array a. Contributions are calculated only when $s_i$ is h. Although a[i] is defined as the number of as starting at $i$, it therefore equals the number starting at $i+1$ at every position where it is used.

The current position contributes:

1

dp[8] * (qpow(2, a[i]) - 1)

If there is no a afterward, the expression in parentheses is $0$, and this h produces no valid choices. The binary-exponentiation result lies in $[1,mod-1]$, so subtracting one here cannot make it negative. Multiplying the two reduced factors also stays within the range of long long.

Why My First Approach Double-Counted

Initially, I maintained a state for the entire pattern: $D_9$ was the cumulative number of ways to choose nunhehheh through the current position:

Then, at each h, I tried to calculate:

1

dp[9] * 2^cntA - 1

There are two independent problems here.

First, I treated a cumulative number of completed matches as the number newly completed at the current position. $D_9$ retains every previously completed nunhehheh; those old choices remain in the state whenever another h appears. Adding the whole of $D_9$ to the answer each time makes old choices contribute repeatedly.

For example:

1

nunhehhehha

Either of the two consecutive hs can be the last character of the fixed prefix, so the correct answer is $2$. If we update the full-pattern state first and then use cumulative $D_9$ at every h, the two contributions include $1$ and $2$ complete prefixes, giving $3$ in total. The extra count is the old choice ending at the first h, counted again at the second h.

The right counting unit is to retain only the cumulative number of matches for the first eight characters, then fix the current h as the ninth. Pairing $D_8$ with the current position gives precisely the choices newly completed at that position.

The second problem is where to subtract one. Suppose that, after fixing the final h, there are $W$ choices for the prefix and $c$ occurrences of a afterward. The choice of taking no a must be excluded for every prefix choice, so the contribution is:

rather than:

The first expression excludes the empty subset once for each prefix choice. The second subtracts only once after combining all choices. They are equal only when $W=1$.

Complexity and Possible Optimization

The length-$8$ DP transition performs a constant number of operations per character, and counting suffix as is also linear. Both parts take $O(n)$ time.

However, the current accepted code calls binary exponentiation at each candidate h. One qpow call takes $O(\log n)$, so for this exact implementation, the worst-case time complexity for a single string is:

The space complexity is $O(n)$, mainly for the suffix-count array; the DP states use only constant space. For multiple test cases, sum the complexities over their respective string lengths.

Precomputing $pow2[k]=2^k\bmod mod$, or maintaining the corresponding power of two during a right-to-left scan, would reduce each power query to $O(1)$ and make the overall time complexity $O(n)$.

Final Accepted Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

#include <bits/stdc++.h>
using namespace std;
using ll = long long;
constexpr int mod = 998244353;

int t;
string temp = "nunhehheh";

ll qpow(ll a, ll b) {
    ll res = 1;
    while (b) {
        if (b & 1) {
            res = res * a % mod;
        }
        a = a * a % mod;
        b >>= 1;
    }
    return res;
}

ll DP(const string &s) {
    ll sum = 0;
    int len = (int) s.length();
    vector<int> a(len + 5, 0);
    vector<ll> dp(9, 0);
    dp[0] = 1;
    for (int i = len - 1; i >= 0; --i) {
        if (i != len - 1) {
            a[i] = a[i + 1];
        }
        if (s[i] == 'a') {
            a[i]++;
        }
    }
    for (int i = 0; i < len; ++i) {
        for (int j = 8; j >= 1; --j) {
            if (s[i] == temp[j - 1]) {
                dp[j] = (dp[j] + dp[j - 1]) % mod;
            }
        }
        if (s[i] == 'h' && dp[8]) {
            sum = (sum + dp[8] * (qpow(2, a[i]) - 1) % mod) % mod;
        }
    }
    return sum;
}

int main() {
    ios::sync_with_stdio(false);
    cin.tie(nullptr);
    cin >> t;
    while (t--) {
        string s;
        cin >> s;
        cout << DP(s) << '\n';
    }
    return 0;
}

The State Granularity I Need to Remember

Fixed-pattern subsequence counting can use one-dimensional DP: $D_j$ counts ways to match the first $j$ characters of the pattern, and the transitions usually need to run in descending order.

But the key here goes beyond applying a template: first decide how to partition the answer. Once choices are classified by the final h of the fixed prefix, the earlier $D_8$, the current position, and the nonempty suffix subset of as account for three disjoint portions of the index selection. This naturally eliminates double counting.

The next time I encounter a structure consisting of a prefix pattern followed by freely chosen suffix characters, I need to ask two questions first: does the current state represent the cumulative count through this position, or only what is newly formed exactly here? And when at least one element must be selected, which quantity should $2^k-1$ multiply? The parentheses here express what is being counted; they are more than a detail of notation.

CCPC 2021 Online H - Subpermutation defines the full-permutation of $n$: concatenate all permutations of $1$ through $n$ in lexicographic order to obtain the sequence $p_n$.

Given $n,m$, count the contiguous subsequences of $p_n$ of length $m$ that are themselves permutations of $1$ through $m$. Answers for multiple test cases are taken modulo $10^9+7$, with:

Each original permutation has length $n$, and the segment length $m$ does not exceed $n$. A segment can therefore meet at most two original permutations, and all cases split completely into two types:

• The segment lies entirely within one permutation;
• The segment takes a nonempty suffix of one permutation followed by a nonempty prefix of the next.

These two types are disjoint, so count them separately and add the results.

Contributions Within One Permutation

A contiguous segment of length $m$ is a permutation of $1$ through $m$ if and only if those $m$ values occupy $m$ consecutive positions in the current permutation.

First choose the starting position of this block, giving $n-m+1$ choices. The values $1$ through $m$ can be arranged arbitrarily inside the block, and the remaining $n-m$ values can also be arranged arbitrarily. The total number of valid segments lying entirely within a single permutation is therefore:

This counts across all $n!$ permutations together. Each permutation satisfying the condition contributes exactly one valid starting position, with no duplication.

Crossing a Permutation Boundary

Fix the length taken from the end of the preceding permutation as $k$, where:

Let $d=m-k$, and split the preceding permutation $P$ by position into:

Here, $|A|=d$, $|B|=n-m$, and $|C|=k$. The segment crossing the boundary consists of $C$ followed by the length-$d$ prefix of the successor permutation $Q$.

First, require only that $A$ and $C$ together contain exactly the values $1$ through $m$. Then:

• The values $1$ through $m$ can occupy the $m$ positions of $A,C$ in any order, giving $m!$ choices;
• The values $m+1$ through $n$ can occupy the middle part $B$ in any order, giving $(n-m)!$ choices.

The number of candidate permutations is:

We still need to determine whether prefix $A$ stays unchanged when moving from $P$ to its lexicographic successor $Q$.

The Subtraction Term from the Longest Decreasing Suffix

To find a permutation’s lexicographic successor, locate its longest decreasing suffix, swap the pivot immediately before that suffix, and put the suffix into its smallest, ascending order.

If $B+C$ is not decreasing, the pivot must lie within $B+C$, so the length-$d$ prefix $A$ remains unchanged. The segment crossing the boundary is then $C+A$, containing each value from $1$ through $m$ exactly once, and is valid.

Conversely, if all of $B+C$ is decreasing, the lexicographic-successor operation crosses the boundary between $A$ and $B+C$. The first $d$ entries of the successor are no longer exactly the original $A$: an element originally to the right of that boundary moves into the prefix. If $P$ is already the last permutation, it has no successor at all. In either case, $C$ and the new prefix can no longer form exactly a permutation of $1$ through $m$.

We therefore only need to subtract the cases where all of $B+C$ is decreasing from the candidate count.

Every value in $B$ is greater than $m$, while every value in $C$ is at most $m$, so their boundary is already decreasing. For all of $B+C$ to be decreasing, it is enough for $B$ and $C$ each to be in descending order:

• Choose the $k$ values of $C$ from $1$ through $m$, giving $\binom{m}{k}$ choices. Once chosen, their descending order in $C$ is unique;
• Arrange the remaining $d$ values arbitrarily in $A$, giving $d!$ choices;
• The descending order of $B$ is unique.

The subtraction term is therefore:

For fixed $k$, the number of valid segments crossing a boundary is:

Summing over all $k$:

Combining the Counts into a Closed Form

Add the contributions from within permutations and across their boundaries:

Ultimately, we need only factorials, inverse factorials, and prefix sums of inverse factorials.

Let:

In the implementation, fac[i], ifac[i], and sum[i] maintain $i!$, $(i!)^{-1}$, and $S_i$, respectively. Each query directly evaluates:

1

return (1ll * n * fac[n - m] % mod - sum[m - 1] + mod) % mod * fac[m] % mod;

All divisions here use modular inverses modulo $10^9+7$. The preprocessing range is below the modulus, so every factorial has an inverse.

When $m=1$, the sum is empty and sum[0]=0. The formula gives $n(n-1)!=n!$, exactly the number of occurrences of 1 in the full sequence. When $n=m$, fac[n-m]=fac[0]=1, so the same formula works without an additional branch.

Complexity and Implementation Boundaries

Precompute factorials, inverse factorials, and prefix sums up to $10^6$. This takes $O(10^6+\log(10^9+7))$ time and $O(10^6)$ space. After preprocessing, each query performs only a constant number of modular operations, taking $O(1)$ per query and $O(T)$ for all queries.

Three implementation boundaries must also remain consistent:

• Set fac[0]=1, so $0!$ is available directly when $n=m$;
• Start the inverse-factorial prefix sum at ifac[1], so the subtraction term is empty when $m=1$;
• For modular subtraction, add mod before taking the remainder to avoid a negative intermediate result.

Final Accepted Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

#include <bits/stdc++.h>
using namespace std;
using ll = long long;
constexpr int mod = 1e9 + 7, maxn = 1e6 + 5;

ll t, fac[maxn], ifac[maxn], sum[maxn];

ll qpow(ll a, ll b) {
    ll res = 1;
    a %= mod;
    while (b) {
        if (b & 1) {
            res = res * a % mod;
        }
        a = a * a % mod;
        b >>= 1;
    }
    return res;
}

void init() {
    fac[0] = 1;
    for (int i = 1; i < maxn; ++i) {
        fac[i] = fac[i - 1] * i % mod;
    }
    ifac[maxn - 1] = qpow(fac[maxn - 1], mod - 2) % mod;
    for (int i = maxn - 2; i >= 0; --i) {
        ifac[i] = ifac[i + 1] * (i + 1) % mod;
    }
    for (int i = 1; i < maxn; ++i) {
        sum[i] = (sum[i - 1] + ifac[i]) % mod;
    }
}

ll ans(int n, int m) {
    return (1ll * n * fac[n - m] % mod - sum[m - 1] % mod + mod) % mod * fac[m] % mod;
}

int main() {
    ios::sync_with_stdio(false);
    cin.tie(nullptr);
    init();
    cin >> t;
    while (t--) {
        int n, m;
        cin >> n >> m;
        cout << ans(n, m) << '\n';
    }
    return 0;
}

CCPC 2021 Online E - Monopoly asks for the earliest arrival time in a process that repeats forever. With direct simulation, there is no clear bound on the number of laps. Only the positions within a single lap form a finite set; every complete lap changes the score by the same cycle sum.

The Problem

The board contains $n$ integers $a_1,a_2,\ldots,a_n$ arranged in a circle. The initial score is $0$. The first step lands on $a_1$, and movement then continues cyclically in order, adding the value at the current position to the score at every step.

Each query gives a target score $x$. Output the number of steps when the score first becomes exactly $x$, or $-1$ if it cannot be reached. The initial state already has score $0$, so the answer for $x=0$ is $0$.

For one test case, $n,m\le 10^5$. Across all test cases, the sums of $n$ and $m$ are each at most $5\times 10^5$. The target score can have absolute value as large as $10^{12}$, so enumerating steps or complete cycles is infeasible.

Positions Within a Cycle

Define the prefix sums of the first lap:

The score change over one complete lap is:

In the implementation, pre[i] stores the within-cycle prefix $P_i$ ($0\le i<n$), while s stores the total $S$ of one complete cycle.

Any step count can be written uniquely as:

Here, $q$ is the number of completed laps, and $i$ is the number of steps already taken in the current lap. The score is:

Keep only $P_0$ through $P_{n-1}$. $P_n=S$ and $P_0$ at the start of the next lap represent the same cycle boundary. Keeping both would give two representations of the same time.

For a query $x$, we now need to find a within-cycle position $i$ satisfying:

while minimizing $q\cdot n+i$.

A Zero Cycle Sum

If $S=0$, a complete lap does not change the score:

Every reachable score already appears in the first lap. For each prefix sum, record only the earliest step where it occurs, then answer queries by looking it up in a hash table. The initial prefix $P_0=0$ corresponds to step $0$.

Later occurrences of the same prefix sum have larger step counts and cannot improve the answer.

f1st stores this mapping from prefix sum to earliest position. When building it, insert a prefix sum only on its first occurrence, so solve1 naturally retrieves the earliest step:

1
2
3

if (!f1st.count(s)) {
    f1st[s] = i;
}

Grouping by Congruence Class

Now consider $S\ne 0$, and let:

Complete cycles change the score by integer multiples of $d$. A necessary condition for a within-cycle prefix $P_i$ to produce the target $x$ is:

Group the prefixes by their nonnegative remainder modulo $d$. For a fixed remainder $r$, write:

Congruence ensures that the required number of laps is an integer. The remaining constraint is $q\ge 0$. The sign of the cycle sum determines which side of the sorted array contains the valid prefixes.

In the implementation, mod normalizes negative remainders to $[0,d)$, and same_rem stores candidate prefixes by remainder. Each candidate’s first component is $P_i$; its second component, val, stores the corresponding step-count adjustment in each of the two cases below.

A Positive Cycle Sum

Here, $S=d$. Substituting into:

gives:

The validity condition is:

The corresponding number of steps is:

For a fixed query, $h\cdot n$ is constant. Sort the prefixes within each congruence class by $P_i$ in ascending order, and maintain prefix minima of:

For a query, binary-search for the last prefix no greater than $x$ and retrieve the minimum through that position.

In the implementation, val = i - k * n stores $i-k_i\cdot n$, and best stores prefix minima after sorting by $P_i$. upper_bound locates the last position satisfying $P_i\le x$, and the answer is h * n + best[rem][idx]. If even the smallest prefix in the group is greater than $x$, there is no valid nonnegative number of laps.

A Negative Cycle Sum

Here, $S=-d$, so the equation becomes:

Therefore:

The validity condition is:

The number of steps can be written as:

We still sort each congruence class by $P_i$ in ascending order, but the valid candidates now form a suffix. Maintain suffix minima of:

For a query, use lower_bound to find the first prefix at least $x$, then retrieve the corresponding suffix minimum.

The adjustment here is val = i + k * n; best stores suffix minima, and the final answer is -h * n + best[rem][idx]. The positive and negative cases share the same congruence grouping; only the direction of the valid range and of the maintained minima changes.

If even the largest prefix in the group is less than $x$, the target is likewise unreachable.

Implementation Details and Boundaries

Taking the remainder of a negative number in C++ may produce a negative remainder. When grouping, always use:

to place remainders in $[0,d)$, so that negative prefixes and negative queries can enter the same group.

A prefix value may occur more than once; deduplication is unnecessary. The prefix or suffix minima automatically retain the candidate with the earlier step count.

For a query $x=0$, output $0$ directly. Even if the score returns to zero later, it cannot do so earlier than the initial state.

Prefix sums, queries, quotients, and expressions for step counts can all exceed the range of int, so use long long throughout.

Complexity

When $S=0$, building the hash table of earliest prefix positions takes $O(n)$ on average, and each query takes $O(1)$ on average.

When $S\ne 0$, sorting all the groups takes $O(n\log n)$ in total, and constructing the minima arrays takes another $O(n)$. Each query first finds its congruence class through a hash table, then binary-searches for the valid range within that group. The average total time complexity is therefore:

The space complexity is $O(n)$.

Accepted Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

#include <bits/stdc++.h>
using namespace std;
using ll = long long;
constexpr int maxn = 1e5 + 5;

int t;

ll mod(ll a, ll b) {
    return (a % b + b) % b;
}

void solve1(int m, unordered_map<ll, int> &f1st) {
    f1st[0] = 0;
    for (int i = 1; i <= m; i++) {
        ll x;
        cin >> x;
        if (f1st.count(x)) {
            cout << f1st[x] << '\n';
        } else {
            cout << -1 << '\n';
        }
    }
}

void solve2(int n, int m, ll s, vector<ll> &pre, unordered_map<ll, vector<pair<ll, ll> > > &same_rem) {
    for (int i = 0; i < n; i++) {
        ll rem = mod(pre[i], abs(s));
        ll k = (pre[i] - rem) / abs(s);
        ll val;
        if (s > 0) {
            val = i - k * n;
        } else {
            val = i + k * n;
        }
        same_rem[rem].push_back({pre[i], val});
    }
    unordered_map<ll, vector<ll> > best;
    for (auto &um: same_rem) {
        ll rem = um.first;
        sort(um.second.begin(), um.second.end());
        int size = um.second.size();
        best[rem] = vector<ll>(size);
        if (s > 0) {
            best[rem][0] = um.second[0].second;
            for (int i = 1; i < size; i++) {
                best[rem][i] = min(best[rem][i - 1], um.second[i].second);
            }
        } else {
            best[rem][size - 1] = um.second[size - 1].second;
            for (int i = size - 2; i >= 0; i--) {
                best[rem][i] = min(best[rem][i + 1], um.second[i].second);
            }
        }
    }
    for (int i = 1; i <= m; i++) {
        ll x;
        cin >> x;
        if (!x) {
            cout << 0 << '\n';
            continue;
        }
        ll d = abs(s);
        ll rem = mod(x, d);
        if (!same_rem.count(rem)) {
            cout << -1 << '\n';
            continue;
        }
        auto &v = same_rem[rem];
        if (s > 0) {
            int idx = upper_bound(v.begin(), v.end(), x, [](ll val, const auto &p) {
                return val < p.first;
            }) - v.begin() - 1;
            if (idx < 0) {
                cout << -1 << '\n';
                continue;
            }
            ll h = (x - rem) / s;
            cout << h * n + best[rem][idx] << '\n';
        } else {
            int idx = lower_bound(v.begin(), v.end(), x, [](const auto &p, ll val) {
                return p.first < val;
            }) - v.begin();
            if (idx == (int) v.size()) {
                cout << -1 << '\n';
                continue;
            }
            ll h = (x - rem) / d;
            cout << -h * n + best[rem][idx] << '\n';
        }
    }
}

void op() {
    int n, m;
    cin >> n >> m;
    ll s = 0;
    vector<int> a(n + 5);
    vector<ll> pre(n + 5, 0);
    unordered_map<ll, int> f1st;
    unordered_map<ll, vector<pair<ll, ll> > > same_rem;
    for (int i = 1; i <= n; i++) {
        cin >> a[i];
        s += a[i];
        if (!f1st.count(s)) {
            f1st[s] = i;
        }
        if (i != n) {
            pre[i] = s;
        }
    }
    if (!s) {
        solve1(m, f1st);
    } else {
        pre[0] = 0;
        solve2(n, m, s, pre, same_rem);
    }
}

int main() {
    ios::sync_with_stdio(false);
    cin.tie(nullptr);
    cin >> t;
    while (t--) {
        op();
    }
    return 0;
}

Until I opened the supposedly finished subtitle file.

It was a mess. Completely unusable.

That was the first time I realized

that the most dangerous part of an Agent workflow

isn’t when it throws an error.

It’s when it doesn’t.

Here’s what happened.

I’d recently wanted to work through a public course on Agentic AI, but the lecture videos were really long. Just watching the recordings wasn’t a very efficient way to study.

So I wondered whether I could turn each video into a Chinese handout that I could pick up and study straight away.

It had also been a while since I’d tinkered with AI Coding.

Somehow, that led me to decide to build a tool myself with Codex.

A tool where I’d drop in a video link and a handout would pop out on its own.

It sounded fairly straightforward. But once I actually broke it down, there were quite a few steps involved.

First, use yt-dlp to download the video and subtitles. If the video had no subtitles, let faster-whisper transcribe it.

Then extract frames with FFmpeg and use Pillow to select the key images.

Finally, align the subtitles with the images, assemble a source package, and hand it to a Writer Agent to generate the Chinese handout.

The whole process looked roughly like this.

Diagram guide: input a lecture video → download video and subtitles (yt-dlp, with faster-whisper when subtitles are unavailable) → extract and select key frames (FFmpeg and Pillow) → align subtitles and images → organize and validate intermediate materials → generate a Chinese handout with the Writer Agent. The left-hand labels mark input, processing, and output.

Because the pipeline was fairly long, I didn’t dare give Codex a single sentence and let it improvise. If any step went off course, everything downstream would follow it.

So I treated Codex as an intern who could get a lot done but still needed supervision.

First, I asked ChatGPT to turn the task into a Plan Prompt, then gave that to Codex to produce an implementation plan.

When the plan came back, I read it myself, then asked ChatGPT to help spot anything missing.

Once there were no obvious problems, I’d generate an Execute Prompt and let Codex actually get to work.

Afterward, Codex also had to submit an execution report.

The process was more or less this.

Diagram guide: GPT generates a Plan Prompt → Codex produces a plan → GPT reviews the plan → GPT generates an Execute Prompt → Codex executes the task. The intern analogy below follows the same sequence: I assign the task, the intern proposes a plan, I review it, I authorize the work, and the intern reports the result.

I also set up a minimal project skeleton by hand first, so it wouldn’t arrive in an empty directory and start improvising immediately.

At the time, I thought I’d broken things down finely enough.

Then the very first round went wrong.

Let’s start with that first round: the subtitle problem from the beginning of this post.

After Codex finished, it reported that the task was complete. The video had been downloaded, and the subtitles had been retrieved.

Normally, I could have moved straight on. But something made me open the subtitle file for a quick look.

I just stared at it.

It hadn’t selected the clean subtitles that came with the video. It had picked the platform’s automatically generated captions instead.

Those automatic captions have a particularly annoying problem: rolling repetition. A sentence finishes, and the next caption repeats it along with the new words. The caption after that repeats it again.

Screenshot guide: the report shows that a platform subtitle track was available, but the code selected selected_language: en and selected_source: automatic. Exact language matching missed a compatible prefixed track key. The sample transcript then shows overlapping, repeated caption fragments, which would contaminate the later content_map and lecture_handout.

If all I’d read was “subtitles retrieved” in the execution report, this problem would never have surfaced.

It might look like nothing more than choosing the wrong subtitle file, but the effect would spread through the pipeline. Content organization, image alignment, and handout generation would all be built on that dirty text.

That’s what makes this kind of problem so troublesome. It doesn’t blow up in front of you. It quietly slips into the final result.

From that round onward, I gave myself a rule.

I couldn’t just take the Agent’s word that it was done.

I had to check the work myself.

After fixing the subtitle problem, I added a fixed step to every round.

Acceptance checks.

Once the Agent finished a task, I wouldn’t move directly to the next round. I’d read its execution report first, then ask ChatGPT to help check for obvious gaps.

If the report looked fine, I’d open a separate, fresh Codex session and give it just one job: check whether the previous round had actually been completed.

This new Agent wasn’t there to continue development. Its only job was to verify the previous round’s work.

Mechanical checks—whether files existed, whether the directories were right, whether the formats were valid—could all go to that Agent.

But questions about content quality and practical value, such as whether a screenshot belonged in the handout or whether the subtitles were really clean, still needed my judgment.

Once both the Agent and I had checked the work, we’d write the lessons from that round back into the project documentation, so the next Agent would have something to remember and wouldn’t repeat the same mistakes.

The process looked roughly like this.

Diagram guide: check the execution report against the actual artifacts and confirm completion → give a separate Agent structured acceptance checks against the goals, scope, and criteria → have a person judge quality, usefulness, practical value, scope violations, incorrect conclusions, and unnatural results → “Solidify” the lessons, rules, and failure examples in reusable project documentation.

But once I started running it, I found another problem.

The acceptance Agent was a little too fond of calling me over.

One moment there was an error here; the next, a missing permission there. Every acceptance run still required me to sit nearby and deal with whatever came up.

That got annoying.

I’d added acceptance checks so I could spend less time hovering. Now I was still hovering, just in a different way. It felt a bit like I’d gone in a circle.

So I added another set of rules to AGENTS.md, specifically for Agent acceptance checks.

When a problem came up, keep going if you can. Don’t dump everything on me at once.

Screenshot guide: the acceptance Agent verifies approved scope and criteria; it does not continue feature development or prove its own implementation successful. The policy distinguishes FAIL, AGENT_FIX_AND_CONTINUE, AGENT_RECORD_AND_CONTINUE, HUMAN_REVIEW_REQUIRED, and BLOCKED. Human review waits until automatic checks finish; a faulty implementation and an unavailable acceptance environment are different outcomes.

During acceptance, I asked the Agent to look first at whether the artifact itself met the requirements.

If it didn’t, there wasn’t much to discuss: send it back for repair. I recorded these cases as FAIL.

If the artifact was fine, the next question was whether the problem blocked the current acceptance run.

If it did, and the Agent couldn’t handle it itself, then it should stop and call me in. Those cases were BLOCKED.

For problems that didn’t block the work, it should handle what it could on its own. If something was worth recording, write it down and keep going.

Questions I had to judge myself, such as screenshot quality, didn’t need to stop the run either. It could finish all the automated checks, then bring them to me together.

The routing rules looked roughly like this.

Diagram guide: does the implementation meet the requirements? If no, return it for repair (FAIL). If yes, does the problem block acceptance? A blocking problem is fixed safely by the Agent when possible; otherwise it is BLOCKED and needs human help with the environment. Nonblocking problems either need human judgment—image quality, duplication, or missing visual changes—or are recorded as warnings or later improvements. Continue automatic checks and summarize the issues in the report.

The later acceptance runs did become much smoother after I added these rules. But soon I hit a subtler problem.

At the key-frame extraction stage, the frame extraction process ran, the output directory was right, and the file formats were normal. As far as the program was concerned, everything was fine.

So the acceptance Agent didn’t flag an error. It only reminded me in the report that I should check screenshot quality manually.

Screenshot guide: no blocking fixes were required. The report suggested later naming and deprecation cleanup, and asked me to inspect the key frames for source authenticity, broken frames, acceptable repetition, and coverage of visible changes within the 180-second smoke-test range. The run was not BLOCKED.

I went through the screenshots myself and did find a problem.

Some images looked different, but were actually successive reveal states of the same slide.

To a program, these two images can look very different. Quite a few elements really have been added.

To a person, though, it’s simply a slide that hasn’t finished revealing its contents yet. Putting that intermediate screenshot into the final handout would just leave the reader confused.

This is hard for a program to judge on its own.

It needs a person.

And that wasn’t even the worst of it. The acceptance Agent also pointed out something else in its report: the last key screenshot stopped at 4110s. The entire lecture was actually 7101s long.

Screenshot guide: max_keyframes_reached: true, with the last key frame at 4110s. The report asks for a manual check of the entire image set and, especially, whether important visual changes are missing from 4110s → 7101s. It suggests raising max_keyframes or adjusting extraction and selection parameters if that later section contains important changes.

That was odd.

Following its suggestion, I went back to the original video and watched the 4110s → 7101s section. The lecturer was still teaching, and the slides were still changing.

So the program hadn’t just missed a few screenshots. It had quietly left out the entire latter part of the lecture.

I was pretty much numb at that point.

Without manual acceptance checks that round, I could have finished the whole project, picked up the handout, and only then discovered that half a lecture was missing. Debugging it afterward would probably have meant several times as much work.

Once I explicitly told the Agent this was a Bug, it found the cause quickly. The candidate-screenshot count limit was simply too low.

After that fix, the first lecture finally worked end to end. I tried a second lecture to see whether the pipeline was actually usable.

The handout hadn’t appeared yet, but the next Bug certainly had.

The project got stuck at video download. It never reached subtitle processing, image extraction, or handout generation.

Screenshot guide: the pipeline stopped at download with TargetResolutionUnavailable; no format met min_video_height, and 1080p was required unless fallback was explicitly allowed. At that point, yt-dlp also reported a missing supported JavaScript runtime, so the report could not yet distinguish an unavailable 1080p source from formats missed by the extraction environment and classified the situation as an environmental blocker.

Eventually, I found that I’d made the video-resolution requirement too rigid: by default, it had to get 1080p. The first lecture happened to have 1080p, so everything worked. The second topped out at 720p, which stopped the entire pipeline.

As I write this, I’m getting a little PTSD.

Because it was exactly like my usual experience solving algorithm problems: the sample cases pass, and all the cases I made up pass too.

Then I submit.

Wrong Answer.

After finishing the project, I looked back over the whole thing.

The next time I ask an Agent to do some work, there are at least 4 more things I’ll check.

First, are the source materials actually right?

If it grabbed the wrong subtitles, nothing downstream can make up for it, however polished the result looks. Garbage in, garbage out.

Second, are the intermediate artifacts actually usable?

Don’t just read the Agent’s execution report. A directory exists, files were generated, and the program didn’t report an error: those things prove it ran. They don’t prove the result is useful.

Third, did the task actually run all the way through?

A few missing screenshots might be hard to notice. Half a missing lecture is a bit much.

Fourth, does it still work on another sample?

Success on the first lecture doesn’t mean success on the second. Passing the samples doesn’t mean you’ve earned an AC—an accepted solution.

After going through all of this, I increasingly feel that acceptance checks may take more skill than I’d imagined.

Agents are well suited to doing the work, and to mechanical checks too.

But what counts as finished, what counts as usable, and what deserves a place in the final result—those are still judgments for a person to make.

I used to think the most important parts were writing Prompts, building workflows, and getting the Agent to run automatically.

Now I tend to think that the person responsible for acceptance is what really sets the ceiling on the whole thing.

AI can help me get more work done.

But I need to know what it means for that work to be done well.

Finally, I put the whole process into one diagram.

Diagram guide: the implementation sequence on the left is Plan Prompt → Codex plan → GPT review → Execute Prompt → Codex execution → results and report. Acceptance on the right adds an initial report review, a planned acceptance run (Validation Plan Prompt → acceptance Agent plan → GPT review), Agent checks, human review, and documentation of lessons. Within Agent checks, the problem-routing rules distinguish FAIL, safe fixes, BLOCKED, human judgment, and recorded nonblocking issues. The bottom arrow returns to implementation for fixes or the next round.

Here’s one of the final handouts as well.

Screenshot guide: this handout section is “Reinforcement Learning and Training Infrastructure,” covering 01:12:08–01:18:38. It combines lecture slides with key points, a detailed explanation, and a short recap. Its main points are that verifiers provide computable feedback, training can optimize around checked outcomes, and reinforcement learning requires infrastructure that coordinates generation, checking, training, and checkpoints.

The project repository is here.

https://github.com/nine19een/video-to-handout

The examples, acceptance workflow, and screenshots in this post all come from this actual development process.

The most troublesome thing about an Agent has never been that it throws an error.

It’s that it doesn’t—and you believe it.

I put together these combinatorics notes while preparing for the Lanqiao Cup National Finals.

It started with ABC458 E. I realized that although I could derive the correct formula, I did not know how to evaluate it. I was unfamiliar with multiplicative inverses, Fermat’s little theorem, division in modular arithmetic, and several common combinatorial counting models. After working through that problem, I followed this thread with a systematic set of practice problems covering binomial-coefficient templates, a linear recurrence for inverses, derangements, the binomial theorem, the reflection principle, and Catalan numbers. (For my ABC458 E solution, see my other post: AtCoder ABC458-E Review: Separators, Nonempty Groups, and Vandermonde’s Convolution.)

This is not an encyclopedia of combinatorics. I will not develop Lucas’s theorem, the extended Euclidean algorithm, the Chinese remainder theorem, or more complicated inclusion–exclusion models. Its scope is clear: collect the combinatorics fundamentals that can turn into contest points.

I therefore focus on three questions:

1. How should division in modular arithmetic actually be handled?
2. How can common combinatorial counting models be translated quickly into formulas?
3. Which implementation details most often cause mistakes with boundaries, modular operations, or intermediate overflow?

Modular Division and Multiplicative Inverses

In ordinary mathematics, division is a natural operation. For example:

In modular arithmetic, however, we cannot simply write:

1

(a / b) % MOD

Integer division truncates, and “divide by $b$” modulo a number should mean “multiply by the multiplicative inverse of $b$,” rather than ordinary division.

Definition of a Multiplicative Inverse

If there is an integer $x$ satisfying:

Note: $\equiv$ means “is congruent to,” which can be understood as equality modulo $MOD$. In other words, $bx$ and $1$ have the same remainder when divided by $MOD$. Equivalently, $bx-1$ is divisible by $MOD$.

then $x$ is called the multiplicative inverse of $b$ modulo $MOD$, written:

Thus:

In code:

1

ans = a * inv(b) % MOD;

Remember that $0$ has no inverse, because no $x$ can satisfy:

The left-hand side is always $0$.

Finding Inverses with Fermat’s Little Theorem

If $MOD$ is prime and $a$ is not a multiple of $MOD$, Fermat’s little theorem gives:

Multiplying both sides by $a^{-1}$ gives:

Therefore, when $MOD$ is prime and $a \not\equiv 0 \pmod {MOD}$:

In code, we usually take its least nonnegative remainder in $[0,MOD-1]$:

The code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

using ll = long long;

//快速幂模板
ll qpow(ll a, ll b) {
    ll res = 1;
    a %= MOD;
    while (b) {
        if (b & 1) {
            res = res * a % MOD;
        }
        a = a * a % MOD;
        b >>= 1;
    }
    return res;
}                           

//费马小定理求逆元
ll inv(ll a) {
    return qpow(a, MOD - 2);
}

The two comments in this template label binary exponentiation and inverse calculation using Fermat’s little theorem, respectively.

Common prime moduli include 1000000007 and 998244353.

Some problems also specify other particular prime moduli.

Conditions for Use

The conditions for using Fermat’s little theorem to find inverses are:

1. $MOD$ is prime;
2. The value $a$ whose inverse is needed is coprime to $MOD$;
3. Under a prime modulus, $a\not\equiv 0\pmod {MOD}$ guarantees that an inverse exists.

If the modulus is not prime, or $a$ is not coprime to it, we cannot blindly use:

1

qpow(a, MOD - 2)

That is a more general inverse problem, usually requiring tools such as the extended Euclidean algorithm. I will not cover it here.

A Linear Recurrence for Inverses of Individual Integers

If we need the inverse of every integer in $1\sim n$, we can of course write this for each $i$:

1

inv[i] = qpow(i, MOD - 2);

But the complexity is:

For large $n$, such as $3\times 10^6$, that is less efficient than we would like.

With a prime modulus, a linear recurrence can find all the inverses of individual integers in $O(n)$ time.

The recurrence starts with:

For $i\ge 2$:

Here, $inv_i$ denotes the multiplicative inverse of $i$ modulo $MOD$.

The point of this section is to understand where the formula comes from.

Why inv[0] Does Not Need Initialization

The meaning of an individual inverse inv[i] is:

When $i=0$, this becomes:

This is clearly impossible, so $0$ has no inverse.

The first meaningful entry in the array is therefore:

1

inv[1] = 1;

because:

Deriving the Recurrence

Suppose we want the inverse of $i$. Integer division gives: