Repository Coding Tasks in GPT-5.4 and Claude Sonnet
On this page
Quick answer: Compare coding agents on the same bounded repository change with the same starting commit, tools, permissions, tests, and review criteria. Flowith did not run the benchmark scores previously shown on this page.
Why the old scorecard was removed
The earlier article assigned aggregate and category scores to GPT-5.4 and Claude Sonnet without publishing repositories, task definitions, outputs, settings, graders, or run logs. Those numbers could not be audited and have been withdrawn.
OpenAI and Anthropic publish provider evaluations for their models. Those results describe disclosed test settings; they do not decide which model will complete your codebase’s tasks with fewer defects.
Define repository tasks
Choose 12 to 24 tasks from real, closed issues or create equivalent fixtures:
- implement an API endpoint with validation;
- repair a concurrency or transaction bug;
- optimize a measured slow query;
- add an authentication or authorization rule;
- migrate a schema with rollback;
- diagnose a production-style failure;
- add tests around legacy behavior;
- refactor without changing public behavior.
Each task needs a clean starting commit, written acceptance criteria, required test commands, forbidden changes, and a time budget. Remove secrets and personal data.
Hold the agent environment constant
Use the exact current model identifiers. Record reasoning settings, context limits, tool access, network access, system instructions, repository state, and any human messages.
Give both candidates the same starting commit and permissions. Run tasks in isolated worktrees or disposable repositories. Do not let the second model see the first model’s patch unless the task explicitly tests review.
Measure completed work
For each run, capture:
| Dimension | Evidence |
|---|---|
| Functional correctness | Acceptance tests |
| Regression safety | Existing suite |
| Code quality | Blinded review findings |
| Security | Threat-specific tests and review |
| Scope control | Unrequested files or behavior changed |
| Recovery | Response to failing tests |
| Human effort | Interventions and review minutes |
| Efficiency | Elapsed time and token/API cost |
A task passes only when its acceptance criteria are met and the patch survives review. A plausible explanation or green self-reported result is not sufficient.
Review without model labels
Have maintainers review anonymized diffs. Record severity and location of defects, then reveal the model after scoring. Keep all attempts, including timeouts and abandoned patches.
Aggregate only comparable tasks. Report pass rate and distributions, not a single decimal score that hides security failures or one catastrophic regression.
Use provider benchmarks as context
OpenAI’s GPT-5.4 release describes its coding and tool-use evaluations. Anthropic’s model documentation identifies current Claude models and capabilities.
Cite those results as provider evidence. Recheck model status and deprecations before running a test; model aliases may change over time.
Decision rule
Choose the model and harness combination that completes more representative tasks with fewer severe review findings and less total human correction, within latency, privacy, and cost constraints. Different task classes may justify different routes.