Linear history with bad commit near end - model identifies second-to-last commit as bad
Tests ability to identify the second-to-last commit as bad. Evaluates bisect near the end of a linear history.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
echo '#!/bin/bash' > test.sh - 05
echo 'grep "ENABLED = true" app.py' >> test.sh - 06
chmod +x test.sh - 07
echo 'ENABLED = true' > app.py - 08
git add test.sh app.py - 09
git commit -m 'Initial commit' - 10
echo '# feature enabled by default' >> app.py - 11
git add app.py - 12
git commit -m 'enable feature' - 13
echo 'ENABLED = false' > app.py - 14
git add app.py - 15
git commit -m 'disable feature' - 16
echo '# add telemetry' >> app.py - 17
git add app.py - 18
git commit -m 'add feature telemetry'
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad.
Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
disable feature
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
6febbb2
f12cd50
JSON Schema
Structured Output
(raw) { "commit": "f12cd50" }
1ea637a
fb41c7f
JSON Schema
Structured Output
(raw) { "commit": "fb41c7f" }
Based on the test results, the build started failing at commit `83c548e disable feature` and continued to fail in the subsequent commit `18d73bb add feature telemetry`.
The bad commit that introduced the bug is:
`83c548e` (or "disable feature")
1a3cf96
JSON Schema
Structured Output
(raw) {"commit": "1a3cf96"}
\boxed{c2493ac}
bdb143f
JSON Schema
Structured Output
(raw) { "commit": "bdb143f" }
abc1234 disable feature
666b075
JSON Schema
Structured Output
(raw) {
"commit": "666b075"
}
Looking at the test results chronologically:
1. **d042ccb** (Initial commit): PASS
2. **794407c** (enable feature): PASS
3. **63dcd11** (disable feature): FAIL ← **First failure**
4. **12ab846** (add feature telemetry): FAIL
The build was passing after `794407c` but failed after `63dcd11`. This indicates that commit `63dcd11` ("disable feature") introduced the bug.
**Answer: 63dcd11**
e1c230d
JSON Schema
Structured Output
(raw) {"commit": "e1c230d"}
Looking at the test results, I can see that:
- The initial commit (73ace31) passes
- Enabling the feature (6b6ffd2) passes
- Disabling the feature (33bc269) fails
- Adding feature telemetry (5a76480) also fails
The bad commit is **33bc269** with the subject "disable feature" - this is the first commit that introduced the failure.
**33bc269**
74e506b
JSON Schema
Structured Output
(raw) {"commit": "74e506b"}