Branching history with bad commit on main branch - model identifies commit before feature merge
Tests ability to identify a bad commit on main before a feature branch merge. Evaluates bisect with branching history.
Baseline Repository
These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.
- 01
git init - 02
git config user.email 'test@test.com' - 03
git config user.name 'Test User' - 04
echo '#!/bin/bash' > test.sh - 05
echo 'grep -v "DEBUG" src/app.js | grep -q "production"' >> test.sh - 06
chmod +x test.sh - 07
mkdir -p src && echo 'const ENV = "production";' > src/app.js - 08
git add test.sh src/app.js && git commit -m 'Initial commit' - 09
echo '// production defaults' >> src/app.js - 10
git add src/app.js && git commit -m 'setup production env' - 11
echo 'const ENV = "development";' > src/app.js - 12
git add src/app.js && git commit -m 'switch to dev' - 13
git checkout -b feature - 14
echo 'const ENV = "development";' > src/app.js - 15
echo 'const DEBUG = true;' >> src/app.js - 16
git add src/app.js && git commit -m 'add diagnostics' - 17
git checkout main - 18
git merge feature
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad.
Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
switch to dev
Campaign Evidence
Loading campaign evidence…
Model Outputs (14)
6be2447
bc2194c
JSON Schema
Structured Output
(raw) {
"commit": "bc2194c"
}
The test results show that commits `4347f60` (Initial commit) and `ff9cb35` (setup production env) both pass. The first failure occurs at commit `a3fd87a` (switch to dev), and all subsequent commits also fail.
This means the bug was introduced in commit `a3fd87a`.
**Answer:** a3fd87a
18c5c90
JSON Schema
Structured Output
(raw) {"commit":"18c5c90"}
Based on the test results, the build was working fine until the commit `e844f3d switch to dev`, which introduced the failure. The subsequent commit `d451fe5 add diagnostics` also failed, but since the failure was already present in `e844f3d`, that is the commit that introduced the bug.
The bad commit is: `e844f3d` (or the subject line "switch to dev").
ba8ec23
JSON Schema
Structured Output
(raw) {"commit": "ba8ec23"}
23166dc
bafb98e
JSON Schema
Structured Output
(raw) {"commit": "bafb98e" }
a6dc74e switch to dev
JSON Schema
Structured Output
(raw) {
"commit": "a6dc74e switch to dev"
}
Based on the test results, I can see that:
1. Commits e9e3627 and 6a19c14 both pass (exit code 0)
2. Commits 3e78b09 and 53942a9 both fail (exit code 1)
The first commit where the tests start failing is **3e78b09** with the subject "switch to dev". This is the commit that introduced the bug.
**Answer: 3e78b09**
c167ff8
JSON Schema
Structured Output
(raw) {"commit": "c167ff8"}
Looking at the test results, I can see that:
- Commits `0608f29` and `8e5e3ff` both pass
- Commits `fd95d61` and `1034857` both fail
Since `fd95d61` is the first commit where the tests start failing, this is the commit that introduced the bug.
**Bad commit: `fd95d61`**
33d612c
JSON Schema
Structured Output
(raw) {"commit": "33d612c"}
609e7bf
Failure: Expected target hash/prefix or subject 'switch to dev'