AIWorth knowingStory 02
Bug Hunt Bench: what AI coding scores really mean
The short answerBug Hunt Bench tests whether AI coding agents can find and fix a known set of defects in two codebases. It scores verified changes, not an agent's confident account of its own work.
The useful part
Why it matters
The useful question is whether a system repairs something, not whether its explanation sounds convincing. For a non-developer, this is a good reason to ask what a benchmark measures before treating a higher number as a better assistant for everything.
Worth doing
Do you need to do anything?
If you use a coding agent, compare its work on a small task you can check. If you do not code, there is no homework here: keep the distinction between a reported success and a verified result.
What does Bug Hunt Bench measure?
The project's method uses 105 planted defects across two repositories. An independent model judges the resulting changes against a hidden answer key. Extra, unplanted fixes are recorded separately rather than added to the main score. A passing test suite alone is not the score.
Why two scores for the same model can differ
Read the agent software and effort setting beside the model name. The repository's 14 September snapshot reports 79 scored runs and 37 planted bugs that no run had fixed. These are dated observations, not permanent properties of the models; consult the live board for later runs.
Our reading rule: compare like with like before comparing winners. Imagine two runners, one timed on a hill and the other on a track. A precise stopwatch does not remove the difference in conditions. Likewise, one coding score should not decide whether you trust a system with a different task.
What the leaderboard cannot decide for you
Two codebases are not your codebase, and finding a known defect is not the same assignment as designing a new feature. We have not rerun this benchmark. Treat its results as evidence about the stated task, then ask what evidence your own decision needs.
Evidence
Primary source
AI-assisted editorial explanation, checked against the linked documentation on 14 September 2026. Examples labelled as illustrations are ours; no independent product test or reproduction is claimed. The original newsletter remains unchanged. Our editorial standards.