AI news · Tools
Seven models, four bugs:
one single difference.
DE Auf Deutsch lesen
Haiku, Sonnet, Opus, Luna, Terra, Sol and Astra get the same bug report, blind and isolated, four times with rising difficulty. What counts isn't what the models claim about themselves, but what a test suite with hidden edge cases actually shows.
python -m pytest after the run, regardless of what a model itself claimed about
the outcome.
Four bugs, one pattern
All four tasks revolve around scheduling conflicts, a domain full of classic edge cases. The difficulty rises from a single wrong comparison operator to a genuine Python pitfall.
| Bug | What it's about | Tests |
|---|---|---|
| 1 · Boundary | Asymmetric comparison (<=/>= instead of </>) falsely flags back-to-back bookings as conflicts | 5 |
| 2 · Normalization | Report mentions only whitespace in room IDs, root cause also affects case sensitivity | 6 |
| 3 · Midnight | Night shifts crossing midnight need a symmetric case split on both sides of the comparison | 6 |
| 4 · Iteration | Classic Python pitfall: a list is mutated while being iterated over, elements get skipped | 5 |
Each task was cross-checked in advance: the broken code deliberately fails the relevant tests, a fix idea that only addresses the reported symptom passes some of them, and only a fix that identifies the actual root cause passes all of them.
The result
| Model | Bug 1 (5) | Bug 2 (6) | Bug 3 (6) | Bug 4 (5) |
|---|---|---|---|---|
| Haiku | 5/5 | 6/6 | 6/6 | 5/5 |
| Sonnet | 5/5 | 6/6 | 6/6 | 5/5 |
| Opus | 5/5 | 6/6 | 6/6 | 5/5 |
| Luna | 5/5 | 6/6 | 6/6 | 5/5 |
| Terra | 5/5 | 6/6 | 6/6 | 5/5 |
| Sol | 5/5 | 6/6 | 0/6 | 5/5 |
| Astra | 5/5 | 6/6 | 6/6 | 5/5 |
27 out of 28 cells green. Six of the seven models solve all four tasks completely and with the same care: they don't just patch the reported case, they recognize the underlying bug class and fix it in general. For bug 2 that means not just stripping whitespace but also normalizing case, even though the bug report only mentions whitespace. For bug 4 it means recognizing the Python pitfall as such, not just catching the two-block special case.
Even Haiku, the smallest and cheapest model in the comparison, found its own, more compact
solution for bug 4 (for block in self.blocks[:], a copy of the list) than Sonnet
and Opus, which both built a separate remaining list — different, but equally
correct.
The one outlier: right idea, broken patch
Sol on bug 3 (midnight wraparound)
gpt-5.6-sol, via codex exec
Sol was the only model to pick a conceptually fully correct approach: split both sides of the comparison into midnight segments and check every pair for overlap — the same approach Sonnet and Opus also chose. But the generated patch has a missing indent:
for new_part_start, new_part_end in new_intervals:
for part_start, part_end in intervals(start, end):
if new_part_start < part_end and new_part_end > part_start:
return True
return True sits at the same indentation level as the if above it,
Python fails at import time with an IndentationError. The code doesn't run, not a
single test can even be collected. What stands out is Sol's own closing remark, unedited from
the log:
That claim is false: Sol's own sandbox genuinely had no access to Python in this run, true for
all four Codex models, because codex exec runs in a PowerShell sandbox without
Python on the path on Windows. The other three Codex models (Luna, Terra, Astra) still wrote
code that actually runs and said so plainly, instead of claiming a "manual check" that
couldn't have happened. The real risk here isn't the missing test environment, it's a false
self-assessment layered on top of it. Which is exactly why nothing in this comparison was
taken on a model's word — every line was independently re-checked with pytest.
A nuance the test suite never asked for
For bug 4 (iteration), Terra and Astra both added a small improvement nobody requested: they
check each existing block against the growing merged range, not against the original new
booking (merged_start/merged_end instead of
new_start/new_end). For the five tests at hand it makes no
difference, but it's the more robust variant for cases this suite doesn't cover: several
initially far-apart blocks that only end up touching each other as the merged range grows
step by step. A detail neither prompt asked for, and one that a pure pass/fail test score
doesn't surface.
What this means for choosing a model
Model size barely matters for clearly scoped logic bugs
For a self-contained bug in a single function, with a clearly worded report, the smallest model here (Haiku) found the actual root cause just as reliably as the largest (Opus). For this kind of task, that favors the cheaper model.
Verification beats self-reporting
The only failure in the whole test would have slipped through on self-reporting alone: Sol claimed its solution worked even though it wasn't even importable. A review that only reads the summary instead of running the code would have missed it.
Real differentiation probably needs a different bug class
Four levels of pure logic bugs in manageable code didn't really weed out any model. Candidates for a harder test: a root cause spread across multiple files, a concurrency issue, or a performance regression that correctness tests don't catch at all.
How reliable is your own AI workflow, really?
I evaluate model and tool choices against real project code, with a real test suite instead of gut feeling. A 30-minute intro call, free of charge.