Agentic Coding Benchmarks: Is That Regression Reall...

Two benchmark scores differ, but so do the memory limits. A score change is not automatically a model regression. Anthropic’s February 2026 infrastructure-noise study investigates resource effects on agentic coding evaluations. The following incident exercise is hypothetical and focuses on interpreting a changed result, rather than reproducing that study.

Describe the runtime as part of the experiment

Record model version, tools, repository state, dependency availability, time budget, concurrency, and network policy. Without this context, a score is not portable. The same model can receive a different opportunity to succeed under another harness.

Separate failure receipts

Classify a run by what actually stopped it: wrong patch, failing test, timeout, preemption, missing dependency, or harness error. Do not collapse every non-pass into “model quality.” This distinction is useful in a take-home postmortem and in a live interview.

Report reproducibility with humility

Rerun a small sample, publish the variance, and state what you did not control. A credible answer can say that a result is directional rather than pretending to have a perfect causal experiment.

Investigate a regression without moving the goalposts

Suppose a nightly evaluation drops after a runner migration. The model identifier and task set appear unchanged. One reaction is to add more retries until the score returns. That can hide the regression: the new run now has more opportunities and a different budget than the baseline.

Instead, choose one failed task and compare receipts. Check the repository commit, container image, CPU and memory limits, test timeout, concurrency, and dependency cache. Verify the actual model configuration too; a familiar public name does not prove that every runtime parameter stayed fixed. Record unavailable details rather than quietly assuming equality.

Separate a patch defect from a blocked execution

For a wrong patch, show the assertion it violates. For a timeout, record which operation was active and whether useful progress was still occurring. For a missing dependency, determine whether network access was intended to be available. These are different hypotheses with different repairs.

A task that requires the agent to work offline should fail when it assumes online access. A harness that accidentally disables a required package mirror creates a different measurement problem. Infrastructure failures remain visible in the end-to-end result; separating them is not permission to remove inconvenient failures from the denominator.

Design a comparison that can change your conclusion

Run the same task under the previous and current resource profiles, holding the agent configuration and reset procedure fixed. Alternate execution order when possible so changing service load does not align with one condition. Repeat enough times to reveal obvious variance, but report the sample size instead of claiming certainty from a few runs.

Suppose the task succeeds only with the old memory limit. Your conclusion is that the resource change affects this workload. You have not established that the model would otherwise be correct on all tasks, or that the higher limit is financially justified. Expand to a relevant sample before changing the production runtime.

Keep both a user-facing completion measure and a diagnostic breakdown. The user experiences a failed task regardless of which layer caused it. The breakdown tells the team whether to fix the patch generator, dependency setup, or resource allocation.

In an interview, finish with a decision: “I would restore a comparable runner for the release comparison, keep the failed runs in the report, and investigate the resource-sensitive cases separately.” That is more useful than simply asking for more compute. A good receipt makes the next experiment smaller and the conclusion more limited.