AIReality checkStory 04
Coding agents miss most maintainer-fixed bugs
What changedPROBE compared frontier coding agents with 2,457 bugs maintainers fixed over 18 months in 20 repositories. Every tested agent recalled under 21% and had under 9% precision against that history. Oddly, evaluators still judged many unmatched findings valid, suggesting agents find a different mix of problems.
The useful part
Why it matters
Repository-wide review needs more than plausible warnings; it needs to find the defects people will actually choose to repair.
Keep in mind
Good to know
Maintainer history is an incomplete reference set, and the validity of extra findings partly depends on model and human judgment.
Evidence