Bad commit changes multiple files (app.py and config.py) - model identifies the multi-file bad commit
Tests ability to identify a multi-file bad commit (app.py + config.py) via bisect. Evaluates bisect for changes spanning multiple files.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '#!/bin/bash' > test.sh
  5. 05 echo 'grep -q "DEBUG = False" app.py && grep -q "DEBUG = False" config.py' >> test.sh
  6. 06 chmod +x test.sh
  7. 07 echo 'DEBUG = False' > app.py && echo 'DEBUG = False' > config.py
  8. 08 git add test.sh app.py config.py && git commit -m 'Initial commit'
  9. 09 echo '# debug defaults' >> config.py
  10. 10 git add app.py config.py && git commit -m 'set debug off'
  11. 11 echo 'DEBUG = True' > app.py && echo 'DEBUG = True' > config.py
  12. 12 git add app.py config.py && git commit -m 'enable debug mode'
  13. 13 echo '# verbose startup logs' >> app.py
  14. 14 git add app.py && git commit -m 'add startup logging'
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad. Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
enable debug mode

Scoped model quality, cost, API time, and token usage for git_bisect/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 283 in → 235 out (16 reasoning)
**Bad commit: `50e1725` — "enable debug mode"** Reasoning: Walking through the test results in chronological order: - `7ef3b9e` (Initial commit): PASS - `3948f0d` (set debug off): PASS - `50e1725` (enable debug mode): **FAIL** ← first failing commit - `7a2dbb1` (add startup logging): FAIL (still broken, inherited from previous commit) The build was passing until `50e1725`, which is the first commit where tests fail. All subsequent commits fail because they build on top of the broken commit. This is exactly how `git bisect` would identify it.
anthropic/claude-fable-5:high__json_schema PASS 100% 500 in → 15 out (0 reasoning)
cba9e28
JSON Schema Structured Output
(raw) {"commit": "cba9e28"}
anthropic/claude-fable-5:low PASS 100% 273 in → 70 out (0 reasoning)
The bad commit is **3ea1f22** ("enable debug mode") — it's the first commit where tests fail (058daba and 89af953 pass, then 3ea1f22 fails and everything after remains broken).
anthropic/claude-fable-5:low__json_schema PASS 100% 502 in → 16 out (0 reasoning)
36ce9ed
JSON Schema Structured Output
(raw) {"commit": "36ce9ed"}
anthropic/claude-fable-5:max PASS 100% 281 in → 449 out (68 reasoning)
**Bad commit: `6875a06` — "enable debug mode"** **Reasoning (bisect-style analysis):** Walking through the test results chronologically: | Commit | Subject | Result | |--------|---------|--------| | b3b5fe4 | Initial commit | ✅ PASS | | c3c7e29 | set debug off | ✅ PASS | | 6875a06 | enable debug mode | ❌ FAIL | | 3493464 | add startup logging | ❌ FAIL | The build transitions from **PASS → FAIL** between `c3c7e29` and `6875a06`. Since `6875a06` ("enable debug mode") is the *first* commit where tests fail, it introduced the bug. The subsequent commit `3493464` fails only because it inherits the broken state — it isn't the root cause.
anthropic/claude-fable-5:max__json_schema PASS 100% 502 in → 127 out (55 reasoning)
774b746
JSON Schema Structured Output
(raw) {"commit": "774b746"}
anthropic/claude-fable-5:medium PASS 100% 275 in → 122 out (0 reasoning)
The bad commit is **acccb1b** — "enable debug mode". Reasoning: Tests pass through `3cef1e6` (set debug off), but the very next commit `acccb1b` (enable debug mode) is the first to fail, and all subsequent commits (`0b1268e`) continue failing. This makes `acccb1b` the commit that introduced the bug.
anthropic/claude-fable-5:medium__json_schema PASS 100% 490 in → 15 out (0 reasoning)
c7f721f
JSON Schema Structured Output
(raw) {"commit": "c7f721f"}
anthropic/claude-fable-5:xhigh PASS 100% 273 in → 226 out (24 reasoning)
**Bad commit: `519f014` — "enable debug mode"** Reasoning (bisect logic): - `e130bfd` (Initial commit): PASS ✅ - `4e1e934` (set debug off): PASS ✅ - `519f014` (enable debug mode): FAIL ❌ ← first failing commit - `3fab881` (add startup logging): FAIL ❌ (inherited breakage) The build transitioned from passing to failing at commit **519f014**, making it the commit that introduced the bug. The subsequent commit `3fab881` fails only because it builds on top of the broken state.
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 498 in → 37 out (15 reasoning)
38d79bb
JSON Schema Structured Output
(raw) {"commit": "38d79bb"}
anthropic/claude-haiku-4.5:high PASS 100% 238 in → 531 out (311 reasoning)
# Analysis Looking at the test results in chronological order: - **34bd9a4** (Initial commit): ✅ PASS - **181135f** (set debug off): ✅ PASS - **02b6873** (enable debug mode): ❌ FAIL ← First failure - **3b42555** (add startup logging): ❌ FAIL The first commit where the build breaks is **02b6873**. The previous commit passed all tests, and this is the first commit to fail, indicating this is where the bug was introduced. ## Answer **02b6873** (or "enable debug mode")
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 406 in → 308 out (265 reasoning)
e97f85e
JSON Schema Structured Output
(raw) {"commit": "e97f85e"}
anthropic/claude-haiku-4.5:low PASS 100% 234 in → 401 out (209 reasoning)
# Analysis Looking at the test results in chronological order: 1. **1bd82f6** (Initial commit): ✅ PASS 2. **808826e** (set debug off): ✅ PASS 3. **6d3992f** (enable debug mode): ❌ FAIL ← First failure 4. **12639ff** (add startup logging): ❌ FAIL (inherits the broken state) The first commit where the build breaks is **6d3992f**, which means this is the commit that introduced the bug. **Answer: `6d3992f` or `enable debug mode`**
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 410 in → 305 out (233 reasoning)
35016a2
JSON Schema Structured Output
(raw) {"commit": "35016a2"}
anthropic/claude-haiku-4.5:medium PASS 100% 242 in → 376 out (191 reasoning)
# Analysis Looking at the test results in chronological order: - **b23c12a** (Initial commit): PASS ✓ - **7ae50de** (set debug off): PASS ✓ - **584aee1** (enable debug mode): **FAIL ✗** ← First failure - **ec0e4d7** (add startup logging): FAIL ✗ (cascading failure) The tests pass through commit `7ae50de`, but fail starting with commit `584aee1`. This is where the build is first broken. **Answer: `584aee1`** or **"enable debug mode"**
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 402 in → 268 out (238 reasoning)
5633843
JSON Schema Structured Output
(raw) {"commit": "5633843"}
anthropic/claude-haiku-4.5:none PASS 100% 217 in → 144 out (0 reasoning)
# Analysis Looking at the test results: - **7bfa114** (Initial commit): ✅ PASS - **4f4d30f** (set debug off): ✅ PASS - **9c96952** (enable debug mode): ❌ FAIL - **d4e4bf4** (add startup logging): ❌ FAIL The first failing test is at commit **9c96952**. Since this is the first commit where the tests fail, this is where the bug was introduced. **Answer: `9c96952`** or **"enable debug mode"**
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 386 in → 12 out (0 reasoning)
4351d16
JSON Schema Structured Output
(raw) {"commit": "4351d16"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 238 in → 472 out (276 reasoning)
# Analysis Looking at the test results in chronological order: 1. **46130e0** (Initial commit): ✅ PASS 2. **41ca599** (set debug off): ✅ PASS 3. **9daa06c** (enable debug mode): ❌ FAIL ← First failure 4. **6d7649f** (add startup logging): ❌ FAIL The first commit where tests fail is **9daa06c**, which is the bad commit that introduced the bug. **Answer: `9daa06c`** (or "enable debug mode")
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 412 in → 376 out (296 reasoning)
84cff13
JSON Schema Structured Output
(raw) {"commit": "84cff13"}
anthropic/claude-opus-4.6:high PASS 100% 211 in → 28 out (0 reasoning)
The bad commit is **7505bc0** ("enable debug mode"). It is the first commit where the test fails.
anthropic/claude-opus-4.6:high__json_schema PASS 100% 383 in → 11 out (0 reasoning)
a61e934
JSON Schema Structured Output
(raw) {"commit":"a61e934"}
anthropic/claude-opus-4.6:low PASS 100% 207 in → 9 out (0 reasoning)
6f6ae05
anthropic/claude-opus-4.6:low__json_schema PASS 100% 379 in → 11 out (0 reasoning)
91627e3
JSON Schema Structured Output
(raw) {"commit":"91627e3"}
anthropic/claude-opus-4.6:max PASS 100% 205 in → 294 out (89 reasoning)
**Bad commit: `77142be` ("enable debug mode")** **Reasoning:** Looking at the test results in chronological order: | Commit | Result | |--------|--------| | `3e23312` Initial commit | ✅ PASS | | `4dd59df` set debug off | ✅ PASS | | `77142be` enable debug mode | ❌ FAIL | | `ae5b7ab` add startup logging | ❌ FAIL | The build was passing up through `4dd59df`, and the first failure occurred at `77142be`. Once a bug is introduced, all subsequent commits continue to fail (as expected). This means **`77142be`** is the commit that broke the build.
anthropic/claude-opus-4.6:max__json_schema PASS 100% 377 in → 78 out (48 reasoning)
06e526e
JSON Schema Structured Output
(raw) {"commit":"06e526e"}
anthropic/claude-opus-4.6:medium PASS 100% 215 in → 9 out (0 reasoning)
e8cd3d0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 381 in → 10 out (0 reasoning)
701fb88
JSON Schema Structured Output
(raw) {"commit":"701fb88"}
anthropic/claude-opus-4.6:none PASS 100% 211 in → 41 out (0 reasoning)
The bad commit is **d863a04** ("enable debug mode"). This is the first commit where the test fails, while the previous commit (`a80157d`) still passes.
anthropic/claude-opus-4.6:none__json_schema PASS 100% 381 in → 12 out (0 reasoning)
e5ab56e
JSON Schema Structured Output
(raw) {"commit":"e5ab56e"}
anthropic/claude-opus-4.6:xhigh PASS 100% 217 in → 9 out (0 reasoning)
`50e1725`
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 373 in → 11 out (0 reasoning)
86e62cf
JSON Schema Structured Output
(raw) {"commit":"86e62cf"}
anthropic/claude-opus-4.7:high PASS 100% 280 in → 9 out (0 reasoning)
b6c9148
anthropic/claude-opus-4.7:high__json_schema PASS 100% 497 in → 17 out (0 reasoning)
1fd8524
JSON Schema Structured Output
(raw) {"commit":"1fd8524"}
anthropic/claude-opus-4.7:low PASS 100% 284 in → 11 out (0 reasoning)
0ca43bf
anthropic/claude-opus-4.7:low__json_schema PASS 100% 503 in → 16 out (0 reasoning)
bdca652
JSON Schema Structured Output
(raw) {"commit":"bdca652"}
anthropic/claude-opus-4.7:max PASS 100% 288 in → 333 out (39 reasoning)
Looking at the test results chronologically: - `84f64f6` Initial commit: **PASS** - `54b8b77` set debug off: **PASS** - `4ac67de` enable debug mode: **FAIL** ← First failing commit - `f8ad3e5` add startup logging: **FAIL** The bad commit is **`4ac67de`** ("enable debug mode"), as it's the first commit where the tests began failing. The subsequent commit (`f8ad3e5`) inherits the failure from this one.
anthropic/claude-opus-4.7:max__json_schema PASS 100% 505 in → 18 out (0 reasoning)
9b69c9b
JSON Schema Structured Output
(raw) {"commit":"9b69c9b"}
anthropic/claude-opus-4.7:medium PASS 100% 284 in → 10 out (0 reasoning)
e2dd4df
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 497 in → 16 out (0 reasoning)
0750b41
JSON Schema Structured Output
(raw) {"commit":"0750b41"}
anthropic/claude-opus-4.7:none PASS 100% 282 in → 8 out (0 reasoning)
f233e22
anthropic/claude-opus-4.7:none__json_schema PASS 100% 499 in → 15 out (0 reasoning)
866d958
JSON Schema Structured Output
(raw) {"commit":"866d958"}
anthropic/claude-opus-4.7:xhigh PASS 100% 280 in → 7 out (0 reasoning)
b631254
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 503 in → 17 out (0 reasoning)
b490dfd
JSON Schema Structured Output
(raw) {"commit":"b490dfd"}
anthropic/claude-opus-4.8:high PASS 100% 275 in → 180 out (18 reasoning)
The bad commit is **6f0d971** ("enable debug mode"). **Reasoning:** - `5178924` (Initial commit): PASS - `e709f0d` (set debug off): PASS - `6f0d971` (enable debug mode): **FAIL** ← first failure - `0d5c475` (add startup logging): FAIL (inherits the broken state) The transition from PASS to FAIL happens at commit `6f0d971`, making it the commit that introduced the bug.
anthropic/claude-opus-4.8:high__json_schema PASS 100% 494 in → 13 out (0 reasoning)
777703a
JSON Schema Structured Output
(raw) {"commit": "777703a"}
anthropic/claude-opus-4.8:low PASS 100% 273 in → 7 out (0 reasoning)
61f0de2
anthropic/claude-opus-4.8:low__json_schema PASS 100% 504 in → 14 out (0 reasoning)
01b2331
JSON Schema Structured Output
(raw) {"commit":"01b2331"}
anthropic/claude-opus-4.8:max PASS 100% 275 in → 666 out (76 reasoning)
# Bad Commit Analysis **Bad commit: `098e07d` — "enable debug mode"** ## Reasoning Looking at the test results in chronological order (oldest first): | Commit | Subject | Result | |--------|---------|--------| | d226f66 | Initial commit | ✅ PASS | | 7d27f9d | set debug off | ✅ PASS | | **098e07d** | **enable debug mode** | ❌ **FAIL** | | 758e6f2 | add startup logging | ❌ FAIL | The build was **passing** through `7d27f9d`, then **first failed** at `098e07d`. Since `758e6f2` also fails (it just inherited the broken state from its parent), the transition point—the commit that actually introduced the bug—is: **`098e07d` ("enable debug mode")** This is the classic "first bad commit" that a `git bisect` would converge on: the earliest commit where the status flips from PASS → FAIL.
anthropic/claude-opus-4.8:max__json_schema PASS 100% 504 in → 107 out (56 reasoning)
4614adb
JSON Schema Structured Output
(raw) {"commit": "4614adb"}
anthropic/claude-opus-4.8:medium PASS 100% 277 in → 185 out (18 reasoning)
The bad commit is **`3f83d74`** ("enable debug mode"). **Reasoning:** - `f6acef6` (Initial commit): PASS - `21caf2e` (set debug off): PASS - `3f83d74` (enable debug mode): **FAIL** ← first failure - `9df85ed` (add startup logging): FAIL (inherits the broken state) The build transitions from PASS to FAIL exactly at `3f83d74`, making it the commit that introduced the bug.
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 500 in → 14 out (0 reasoning)
db778d7
JSON Schema Structured Output
(raw) {"commit": "db778d7"}
anthropic/claude-opus-4.8:none PASS 100% 275 in → 102 out (0 reasoning)
The bad commit is **b0b81f5** ("enable debug mode"). This is the first commit where the tests transition from PASS to FAIL. The preceding commit (265986d) passed, and b0b81f5 is where the failure was introduced, with the failure persisting into the subsequent commit (7098fc6).
anthropic/claude-opus-4.8:none__json_schema PASS 100% 506 in → 16 out (0 reasoning)
2b4ccc4
JSON Schema Structured Output
(raw) {"commit": "2b4ccc4"}
anthropic/claude-opus-4.8:xhigh PASS 100% 275 in → 327 out (42 reasoning)
The bad commit is **`28dcb51`** — "enable debug mode". **Reasoning:** Looking at the test results in chronological order (oldest to newest): | Commit | Subject | Result | |--------|---------|--------| | cf87eab | Initial commit | ✅ PASS | | eafdefc | set debug off | ✅ PASS | | **28dcb51** | **enable debug mode** | ❌ **FAIL** | | 11efb7b | add startup logging | ❌ FAIL | The transition from PASS to FAIL occurs at commit **`28dcb51`**. The commit immediately before it (`eafdefc`) still passed, and every commit from `28dcb51` onward fails. This makes `28dcb51` the first bad commit that introduced the bug.
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 498 in → 14 out (0 reasoning)
03978b0
JSON Schema Structured Output
(raw) {"commit": "03978b0"}
anthropic/claude-sonnet-4.6:high PASS 100% 209 in → 12 out (0 reasoning)
The bad commit is **d737501**
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 391 in → 12 out (0 reasoning)
0d96a87
JSON Schema Structured Output
(raw) {"commit":"0d96a87"}
anthropic/claude-sonnet-4.6:low PASS 100% 209 in → 10 out (0 reasoning)
`5a8a493`
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 381 in → 11 out (0 reasoning)
3659cba
JSON Schema Structured Output
(raw) {"commit":"3659cba"}
anthropic/claude-sonnet-4.6:max PASS 100% 213 in → 339 out (108 reasoning)
## Analysis Looking at the test results in chronological order: | Commit | Subject | Result | |--------|---------|--------| | `fc626d3` | Initial commit | ✅ PASS | | `d1ccc64` | set debug off | ✅ PASS | | `44517f1` | enable debug mode | ❌ FAIL | | `fcd9984` | add startup logging | ❌ FAIL | The build transitions from **PASS → FAIL** at commit `44517f1`. All subsequent commits also fail, indicating the bug was introduced here and never fixed. ## Bad Commit **`44517f1`** The commit "enable debug mode" introduced the breaking change. The subsequent commit `fcd9984` ("add startup logging") also fails, but it inherited the broken state rather than causing it.
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 383 in → 95 out (56 reasoning)
a5115c5
JSON Schema Structured Output
(raw) {"commit":"a5115c5"}
anthropic/claude-sonnet-4.6:medium PASS 100% 217 in → 8 out (0 reasoning)
e64e45d
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 377 in → 11 out (0 reasoning)
aad7337
JSON Schema Structured Output
(raw) {"commit":"aad7337"}
anthropic/claude-sonnet-4.6:none PASS 100% 209 in → 14 out (0 reasoning)
The bad commit is **c16bbf5**
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 377 in → 10 out (0 reasoning)
12975fd
JSON Schema Structured Output
(raw) {"commit":"12975fd"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 213 in → 54 out (0 reasoning)
Based on the test results, the build was passing up through commit `b5b0e78` and started failing at commit `40042e1`. The bad commit is: **`40042e1`** ("enable debug mode")
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 385 in → 11 out (0 reasoning)
21b096d
JSON Schema Structured Output
(raw) {"commit":"21b096d"}
anthropic/claude-sonnet-5:high PASS 100% 277 in → 14 out (0 reasoning)
27b9c5e enable debug mode
anthropic/claude-sonnet-5:high__json_schema PASS 100% 500 in → 14 out (0 reasoning)
38967f7
JSON Schema Structured Output
(raw) {"commit": "38967f7"}
anthropic/claude-sonnet-5:low PASS 100% 277 in → 15 out (0 reasoning)
e35a542 (enable debug mode)
anthropic/claude-sonnet-5:low__json_schema PASS 100% 496 in → 15 out (0 reasoning)
4c670cd
JSON Schema Structured Output
(raw) {"commit":"4c670cd"}
anthropic/claude-sonnet-5:medium PASS 100% 279 in → 103 out (0 reasoning)
**4a02b33** — "enable debug mode" This is the first commit where the build failed (exit code 1), while the preceding commit (6cbeec9) still passed. Since 579b74c continues to fail, it's consistent with the bug being introduced at 4a02b33 rather than fixed or newly broken later.
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 486 in → 14 out (0 reasoning)
08631bb
JSON Schema Structured Output
(raw) {"commit":"08631bb"}
anthropic/claude-sonnet-5:none PASS 100% 279 in → 15 out (0 reasoning)
a6e2c0d enable debug mode
anthropic/claude-sonnet-5:none__json_schema PASS 100% 502 in → 15 out (0 reasoning)
ca9c30c
JSON Schema Structured Output
(raw) {"commit": "ca9c30c"}
anthropic/claude-sonnet-5:xhigh PASS 100% 279 in → 322 out (25 reasoning)
**Bad commit: `9fbd1e1` - "enable debug mode"** **Reasoning:** Looking at the test results in chronological order: - `905bda9` (Initial commit): PASS - `aa287ca` (set debug off): PASS - `9fbd1e1` (enable debug mode): **FAIL** ← first failure - `089b4fe` (add startup logging): FAIL (still broken) The build was passing through `aa287ca` and first failed at `9fbd1e1`. Since `089b4fe` also fails, it's possible that commit either didn't fix the issue or introduced unrelated changes on top of an already-broken build — but the regression was clearly introduced by `9fbd1e1`, which is the first commit where the test suite fails. Using bisection logic (last good → first bad), `9fbd1e1` is the culprit.
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 494 in → 20 out (0 reasoning)
39b143a enable debug mode
JSON Schema Structured Output
(raw) {"commit":"39b143a enable debug mode"}
arcee-ai/trinity-large-thinking:high PASS 100% 194 in → 345 out (341 reasoning)
e5734fd
arcee-ai/trinity-large-thinking:low PASS 100% 194 in → 472 out (374 reasoning)
The bad commit is the one that introduced the bug, which is the first commit where the test fails. According to the test results, the test fails starting at commit `f94f590` ("enable debug mode"). The subsequent commit `7cf32d0` also fails because it is built on top of the bad commit. Thus, the bad commit hash (first 7 characters) is **f94f590** or the subject line **"enable debug mode"**.
arcee-ai/trinity-large-thinking:medium PASS 100% 194 in → 268 out (261 reasoning)
49c27b9
arcee-ai/trinity-large-thinking:xhigh PASS 100% 200 in → 356 out (350 reasoning)
4bbb9e8
arcee-ai/trinity-mini:high PASS 100% 196 in → 916 out (634 reasoning)
3e09181 **Reasoning:** 1. **Test Results Order:** The test results are listed oldest first. This means: * `f365a6d` (Initial commit): PASS * `b6d5ea1` (set debug off): PASS * `3e09181` (enable debug mode): FAIL * `e130804` (add startup logging): FAIL 2. **Identifying the First Failure:** The first commit in the history that causes a test failure is `3e09181` ("enable debug mode"). This is the commit where the build broke for the first time. 3. **Why it's the Bad Commit:** The problem states there is "one bad commit that broke the build". The test results show that the build was working correctly (`PASS`) up to and including `b6d5ea1` ("set debug off"). The very next commit, `3e09181` ("enable debug mode"), caused the build to fail (`FAIL`). Therefore, `3e09181` is the commit that introduced the bug. 4. **Why `e130804` is Not the Bad Commit:** While `e130804` ("add startup logging") also fails, it fails *after* the build was already broken by `3e09181`. It did not introduce the initial bug; it likely exacerbated the existing problem or introduced a separate issue. The problem specifies the commit that "broke the build", meaning the first commit that caused the failure. That is `3e09181`.
arcee-ai/trinity-mini:high__json_schema PASS 100% 196 in → 1,213 out (1,221 reasoning)
b0f383c enable debug mode
JSON Schema Structured Output
(raw) { "commit":"b0f383c enable debug mode" }
arcee-ai/trinity-mini:low PASS 100% 200 in → 439 out (416 reasoning)
b268edd
arcee-ai/trinity-mini:low__json_schema PASS 100% 194 in → 745 out (672 reasoning)
6d41e6d
JSON Schema Structured Output
(raw) { "commit": "6d41e6d" }
arcee-ai/trinity-mini:medium PASS 100% 192 in → 464 out (519 reasoning)
63c140c
arcee-ai/trinity-mini:medium__json_schema PASS 100% 194 in → 304 out (331 reasoning)
376adde
JSON Schema Structured Output
(raw) { "commit": "376adde" }
arcee-ai/trinity-mini:xhigh PASS 100% 192 in → 836 out (779 reasoning)
1e258cb
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 196 in → 662 out (578 reasoning)
45cac4a
JSON Schema Structured Output
(raw) { "commit": "45cac4a" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 187 in → 140 out (132 reasoning)
aa73286
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 185 in → 11 out (0 reasoning)
b756384
JSON Schema Structured Output
(raw) {"commit": "b756384"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 189 in → 353 out (347 reasoning)
aba1d7f
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 189 in → 242 out (229 reasoning)
1f48648
JSON Schema Structured Output
(raw) {"commit": "1f48648"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 287 in → 145 out (139 reasoning)
80b83a4
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 262 in → 163 out (144 reasoning)
1b5cb23
JSON Schema Structured Output
(raw) { "commit": "1b5cb23" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 189 in → 194 out (187 reasoning)
93a3c88
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 187 in → 229 out (200 reasoning)
5266182
JSON Schema Structured Output
(raw) { "commit": "5266182" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 197 in → 41 out (0 reasoning)
The bad commit is **6b8c36e** (enable debug mode), as it's the first commit where the test fails (exit code 1), and all subsequent commits also fail.
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 191 in → 14 out (0 reasoning)
30668b0
JSON Schema Structured Output
(raw) { "commit": "30668b0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 258 in → 106 out (106 reasoning)
337823d
deepseek/deepseek-v4-flash:high PASS 100% 183 in → 131 out (121 reasoning)
234cb14
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 191 in → 120 out (107 reasoning)
a7f2bd9
JSON Schema Structured Output
(raw) {"commit": "a7f2bd9"}
deepseek/deepseek-v4-flash:low PASS 100% 189 in → 163 out (158 reasoning)
81aba91
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 195 in → 95 out (82 reasoning)
5dd8454
JSON Schema Structured Output
(raw) { "commit": "5dd8454" }
deepseek/deepseek-v4-flash:medium PASS 100% 191 in → 186 out (179 reasoning)
cac30e5
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 195 in → 157 out (141 reasoning)
f14a9a3
JSON Schema Structured Output
(raw) { "commit": "f14a9a3" }
deepseek/deepseek-v4-flash:none PASS 100% 191 in → 40 out (0 reasoning)
The bad commit is `7d061f8` (enable debug mode), as this is the first commit where the test fails (exit code 1), and all subsequent commits also fail.
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 259 in → 10 out (0 reasoning)
3668f37
JSON Schema Structured Output
(raw) {"commit": "3668f37"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 187 in → 180 out (177 reasoning)
c9bc465
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 189 in → 128 out (110 reasoning)
8f4f67b
JSON Schema Structured Output
(raw) { "commit": "8f4f67b" }
deepseek/deepseek-v4-pro:high PASS 100% 193 in → 277 out (271 reasoning)
7524f14
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 187 in → 262 out (252 reasoning)
947fe83
JSON Schema Structured Output
(raw) {"commit": "947fe83"}
deepseek/deepseek-v4-pro:low PASS 100% 185 in → 70 out (0 reasoning)
Based on the test results, the bad commit is: **cabbc79** or the subject line: **enable debug mode** This is the first commit where the test fails (exit code 1), and since it failed on this commit and all subsequent commits, it's clear that enabling debug mode introduced the bug that broke the build.
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 392 in → 285 out (273 reasoning)
2d8085f
JSON Schema Structured Output
(raw) {"commit": "2d8085f"}
deepseek/deepseek-v4-pro:medium PASS 100% 183 in → 231 out (226 reasoning)
babce01
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 191 in → 227 out (216 reasoning)
7853b1b
JSON Schema Structured Output
(raw) {"commit":"7853b1b"}
deepseek/deepseek-v4-pro:none PASS 100% 191 in → 58 out (0 reasoning)
The bad commit is **2477cec enable debug mode**. This is the first commit where the test fails. The commits before it (98f93d4 and 8a2e230) pass, and the failure continues into the subsequent commit (8bd4986).
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 382 in → 8 out (0 reasoning)
de78558
JSON Schema Structured Output
(raw) {"commit":"de78558"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 268 in → 303 out (298 reasoning)
6334689
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 274 in → 401 out (385 reasoning)
52a3f9e
JSON Schema Structured Output
(raw) { "commit": "52a3f9e" }
google/gemini-3-flash-preview:high PASS 100% 215 in → 223 out (217 reasoning)
1059dd7
google/gemini-3-flash-preview:high__json_schema PASS 100% 270 in → 172 out (162 reasoning)
0115da1
JSON Schema Structured Output
(raw) {"commit":"0115da1"}
google/gemini-3-flash-preview:low PASS 100% 215 in → 266 out (259 reasoning)
7a9a1a2
google/gemini-3-flash-preview:low__json_schema PASS 100% 268 in → 188 out (179 reasoning)
afe62df
JSON Schema Structured Output
(raw) {"commit": "afe62df"}
google/gemini-3-flash-preview:medium PASS 100% 207 in → 273 out (268 reasoning)
05ed8bd
google/gemini-3-flash-preview:medium__json_schema PASS 100% 210 in → 311 out (301 reasoning)
a3edc2a
JSON Schema Structured Output
(raw) {"commit":"a3edc2a"}
google/gemini-3-flash-preview:xhigh PASS 100% 205 in → 256 out (239 reasoning)
The bad commit is **0e9eac5** (enable debug mode).
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 206 in → 214 out (203 reasoning)
3c96cb9
JSON Schema Structured Output
(raw) {"commit": "3c96cb9"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 212 in → 639 out (566 reasoning)
The bad commit is **7dbb851** (enable debug mode). **Reasoning:** The test results show that the build was passing in commit `25ad2e3`, but failed starting with commit `7dbb851`. Therefore, `7dbb851` is the commit that introduced the bug.
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 272 in → 307 out (297 reasoning)
feb270e
JSON Schema Structured Output
(raw) {"commit": "feb270e"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 209 in → 251 out (135 reasoning)
The bad commit is **3fe8620**. **Reasoning:** * The commit `226b578` passed (exit code 0), meaning the build was healthy at that point. * The next commit `3fe8620` failed (exit code 1), meaning the bug was introduced in this commit. * The subsequent commit `11fcde7` also failed, but because the failure began at `3fe8620`, that is the commit that introduced the bug.
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 218 in → 126 out (109 reasoning)
4567358
JSON Schema Structured Output
(raw) { "commit": "4567358" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 211 in → 238 out (221 reasoning)
The bad commit is **593fd02** (enable debug mode).
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 214 in → 179 out (167 reasoning)
075e950
JSON Schema Structured Output
(raw) {"commit": "075e950"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 216 in → 564 out (485 reasoning)
The bad commit is **992191a** (enable debug mode). **Reasoning:** The test results show that `54fdf0b` passed, but the very next commit in the history, `992191a`, caused the build to fail. Therefore, the breakage was introduced in `992191a`.
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 212 in → 569 out (559 reasoning)
84bf90f
JSON Schema Structured Output
(raw) {"commit":"84bf90f"}
google/gemini-3.1-pro-preview:high PASS 100% 203 in → 199 out (193 reasoning)
aeb28f3
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 268 in → 253 out (243 reasoning)
a4aadaf
JSON Schema Structured Output
(raw) {"commit": "a4aadaf"}
google/gemini-3.1-pro-preview:low PASS 100% 209 in → 126 out (105 reasoning)
The bad commit that introduced the bug is **c4aacca** (or "enable debug mode").
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 270 in → 170 out (158 reasoning)
62d00e1
JSON Schema Structured Output
(raw) {"commit": "62d00e1"}
google/gemini-3.1-pro-preview:medium PASS 100% 213 in → 249 out (138 reasoning)
The bad commit that introduced the bug is **96626d2** (or "enable debug mode"). Looking at the test results from oldest to newest: * bca9b37 Initial commit - PASS * 439eaf5 set debug off - PASS * **96626d2 enable debug mode - FAIL** *(This is the first commit where the test failed, meaning this commit introduced the bug)* * d75a4b5 add startup logging - FAIL
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 270 in → 99 out (88 reasoning)
64a6ce2
JSON Schema Structured Output
(raw) {"commit": "64a6ce2"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 209 in → 154 out (147 reasoning)
90730b7
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 272 in → 132 out (122 reasoning)
4ce9911
JSON Schema Structured Output
(raw) {"commit":"4ce9911"}
google/gemini-3.5-flash-lite:high PASS 100% 213 in → 381 out (374 reasoning)
1363f06
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 218 in → 375 out (358 reasoning)
9c3f6d8
JSON Schema Structured Output
(raw) { "commit": "9c3f6d8" }
google/gemini-3.5-flash-lite:low PASS 100% 213 in → 98 out (92 reasoning)
1a2f8bf
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 272 in → 16 out (0 reasoning)
4cf217a
JSON Schema Structured Output
(raw) { "commit": "4cf217a" }
google/gemini-3.5-flash-lite:max PASS 100% 211 in → 335 out (329 reasoning)
21cf753
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 218 in → 317 out (305 reasoning)
111339f
JSON Schema Structured Output
(raw) {"commit": "111339f"}
google/gemini-3.5-flash-lite:medium PASS 100% 211 in → 195 out (188 reasoning)
120256b
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 274 in → 290 out (278 reasoning)
7d1778c
JSON Schema Structured Output
(raw) {"commit": "7d1778c"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 214 in → 434 out (427 reasoning)
0922724
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 214 in → 520 out (504 reasoning)
ce3a693
JSON Schema Structured Output
(raw) { "commit": "ce3a693" }
google/gemini-3.5-flash:high PASS 100% 209 in → 268 out (249 reasoning)
The commit that introduced the bug is **da81fc4** (enable debug mode).
google/gemini-3.5-flash:high__json_schema PASS 100% 270 in → 262 out (250 reasoning)
d4a50b9
JSON Schema Structured Output
(raw) {"commit": "d4a50b9"}
google/gemini-3.5-flash:low PASS 100% 217 in → 86 out (79 reasoning)
c16227d
google/gemini-3.5-flash:low__json_schema PASS 100% 278 in → 87 out (75 reasoning)
955b54a
JSON Schema Structured Output
(raw) {"commit": "955b54a"}
google/gemini-3.5-flash:medium PASS 100% 211 in → 211 out (205 reasoning)
3160ce8
google/gemini-3.5-flash:medium__json_schema PASS 100% 270 in → 162 out (153 reasoning)
244faa2
JSON Schema Structured Output
(raw) {"commit":"244faa2"}
google/gemini-3.5-flash:xhigh PASS 100% 217 in → 202 out (195 reasoning)
b805f23
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 274 in → 330 out (319 reasoning)
23d63b6
JSON Schema Structured Output
(raw) {"commit":"23d63b6"}
google/gemini-3.6-flash:high PASS 100% 211 in → 196 out (189 reasoning)
9518735
google/gemini-3.6-flash:high__json_schema PASS 100% 216 in → 323 out (312 reasoning)
8a1a63c
JSON Schema Structured Output
(raw) {"commit":"8a1a63c"}
google/gemini-3.6-flash:low PASS 100% 214 in → 128 out (107 reasoning)
The commit that introduced the bug is **f37d8c6** (enable debug mode).
google/gemini-3.6-flash:low__json_schema PASS 100% 272 in → 34 out (19 reasoning)
ce729fc
JSON Schema Structured Output
(raw) { "commit": "ce729fc" }
google/gemini-3.6-flash:max PASS 100% 218 in → 292 out (285 reasoning)
69f7c24
google/gemini-3.6-flash:max__json_schema PASS 100% 216 in → 309 out (298 reasoning)
24f93a6
JSON Schema Structured Output
(raw) {"commit":"24f93a6"}
google/gemini-3.6-flash:medium PASS 100% 210 in → 270 out (254 reasoning)
`661e4d3` (or "enable debug mode")
google/gemini-3.6-flash:medium__json_schema PASS 100% 208 in → 295 out (286 reasoning)
fad397b
JSON Schema Structured Output
(raw) {"commit":"fad397b"}
google/gemini-3.6-flash:xhigh PASS 100% 213 in → 276 out (262 reasoning)
`e80c95f` (enable debug mode)
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 214 in → 241 out (230 reasoning)
a734352
JSON Schema Structured Output
(raw) {"commit":"a734352"}
google/gemma-4-26b-a4b-it:high PASS 100% 231 in → 288 out (192 reasoning)
3e2c9fb
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 227 in → 320 out (205 reasoning)
9573917
JSON Schema Structured Output
(raw) { "commit": "9573917" }
google/gemma-4-26b-a4b-it:low PASS 100% 225 in → 209 out (169 reasoning)
f661c89
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 231 in → 325 out (211 reasoning)
b1e7f7a
JSON Schema Structured Output
(raw) { "commit": "b1e7f7a" }
google/gemma-4-26b-a4b-it:medium PASS 100% 227 in → 226 out (218 reasoning)
722a89a
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 229 in → 363 out (243 reasoning)
3b4c067
JSON Schema Structured Output
(raw) { "commit": "3b4c067" }
google/gemma-4-26b-a4b-it:none PASS 100% 224 in → 8 out (0 reasoning)
495e663
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 222 in → 13 out (0 reasoning)
7678010
JSON Schema Structured Output
(raw) {"commit": "7678010"}
google/gemma-4-26b-a4b-it:xhigh PASS 100% 231 in → 403 out (395 reasoning)
a489571
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 227 in → 261 out (165 reasoning)
54370a2
JSON Schema Structured Output
(raw) { "commit": "54370a2" }
google/gemma-4-31b-it:high PASS 100% 230 in → 184 out (118 reasoning)
9c61b32
google/gemma-4-31b-it:high__json_schema PASS 100% 232 in → 211 out (134 reasoning)
74b83d7
JSON Schema Structured Output
(raw) {"commit": "74b83d7"}
google/gemma-4-31b-it:low PASS 100% 229 in → 161 out (1 reasoning)
ed41b0c
google/gemma-4-31b-it:low__json_schema PASS 100% 232 in → 166 out (101 reasoning)
29644e6
JSON Schema Structured Output
(raw) {"commit": "29644e6"}
google/gemma-4-31b-it:medium PASS 100% 233 in → 151 out (98 reasoning)
e27b149
google/gemma-4-31b-it:medium__json_schema PASS 100% 228 in → 138 out (88 reasoning)
1f646e1
JSON Schema Structured Output
(raw) {"commit": "1f646e1"}
google/gemma-4-31b-it:none PASS 100% 220 in → 7 out (0 reasoning)
de45b17
google/gemma-4-31b-it:none__json_schema PASS 100% 230 in → 13 out (0 reasoning)
a7a9130
JSON Schema Structured Output
(raw) {"commit": "a7a9130"}
google/gemma-4-31b-it:xhigh PASS 100% 224 in → 151 out (104 reasoning)
df0eb18
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 228 in → 181 out (121 reasoning)
6d6df33
JSON Schema Structured Output
(raw) {"commit": "6d6df33"}
ibm-granite/granite-4.1-8b PASS 100% 210 in → 80 out
The test results indicate that the build failed after the commit `03f1fd4 enable debug mode`. Since this is the first commit in the history that caused a failure (`PASS` to `FAIL`), it is the bad commit that introduced the bug. **Bad commit hash (first 7 characters):** `03f1fd4` **Commit subject line:** enable debug mode
ibm-granite/granite-4.1-8b__json_schema PASS 100% 206 in → 13 out
d34a10b
JSON Schema Structured Output
(raw) { "commit": "d34a10b" }
inclusionai/ling-2.6-flash PASS 100% 235 in → 8 out
57c4c72
inclusionai/ling-2.6-flash__json_schema PASS 100% 235 in → 14 out
27fb2f1 enable debug mode
JSON Schema Structured Output
(raw) {"commit":"27fb2f1 enable debug mode"}
liquid/lfm-2-24b-a2b PASS 100% 208 in → 469 out
Let's analyze step-by-step: - **Initial commit (`2d3a7f7`)**: PASS - **`f157449 set debug off`**: PASS - **`6f207dd enable debug mode`**: FAIL - **`6eeede5 add startup logging`**: FAIL The build fails at `6f207dd`, but also fails at `6eeede5`. Since commits are in reverse chronological order, the **bad commit is the earliest one where the failure occurs**—but actually, both `6f207dd` and `6eeede5` fail. However, since `6f207dd` comes before `6eeede5` in the log, it means `6f207dd` was committed after `f157449` and before `6eeede5`. But the **build fails at `6f207dd`**, so that commit is suspect. The next commit (`6eeede5`) also fails, but that could be due to changes from `6f207dd`. However, the **first commit that introduces the bug** is typically the one whose changes cause the failure when applied. Here, `6f207dd` enables debug mode, which might have changed behavior leading to the failure in `6eeede5`. But since `6f207dd` itself causes a FAIL, it's likely the culprit. But let's check the order: - Newest: `6eeede5` - Then: `6f207dd` - Then: `f157449` - Oldest: `2d3a7f7` So the sequence is: ``` 6eeede5 add startup logging (FAIL) 6f207dd enable debug mode (FAIL) f157449 set debug off (PASS) 2d3a7f7 Initial commit (PASS) ``` Since `6f207dd` is the first commit that fails, and it's before `6eeede5`, it's possible that `6f207dd` introduced the bug. **Conclusion:** The bad commit is `6f207dd`. **Answer:** `6f207dd`
minimax/minimax-m2.5:high PASS 100% 223 in → 280 out (273 reasoning)
621d9d9
minimax/minimax-m2.5:high__json_schema PASS 100% 227 in → 319 out (276 reasoning)
3cbf5d7
JSON Schema Structured Output
(raw) { "commit": "3cbf5d7" }
minimax/minimax-m2.5:low PASS 100% 223 in → 294 out (291 reasoning)
bfa1534
minimax/minimax-m2.5:low__json_schema PASS 100% 237 in → 452 out (439 reasoning)
2ba1c96
JSON Schema Structured Output
(raw) { "commit": "2ba1c96" }
minimax/minimax-m2.5:medium PASS 100% 223 in → 282 out (259 reasoning)
e839c2c
minimax/minimax-m2.5:medium__json_schema PASS 100% 223 in → 284 out (263 reasoning)
e056bb0
JSON Schema Structured Output
(raw) { "commit": "e056bb0" }
minimax/minimax-m2.5:xhigh PASS 100% 231 in → 355 out (347 reasoning)
e43c7d1
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 231 in → 169 out (157 reasoning)
876df32
JSON Schema Structured Output
(raw) { "commit": "876df32" }
minimax/minimax-m2.7:high PASS 100% 229 in → 368 out (355 reasoning)
c6a5cdd (enable debug mode)
minimax/minimax-m2.7:high__json_schema PASS 100% 227 in → 16 out (14 reasoning)
a82dfa9
JSON Schema Structured Output
(raw) { "commit": "a82dfa9" }
minimax/minimax-m2.7:low PASS 100% 222 in → 227 out (201 reasoning)
bb03b9e (enable debug mode) is the first commit that breaks the build.
minimax/minimax-m2.7:low__json_schema PASS 100% 356 in → 398 out (388 reasoning)
269c11c
JSON Schema Structured Output
(raw) {"commit":"269c11c"}
minimax/minimax-m2.7:medium PASS 100% 223 in → 385 out (372 reasoning)
7a16499
minimax/minimax-m2.7:medium__json_schema PASS 100% 360 in → 413 out (398 reasoning)
33a2cc1
JSON Schema Structured Output
(raw) { "commit": "33a2cc1" }
minimax/minimax-m2.7:xhigh PASS 100% 207 in → 423 out (283 reasoning)
Looking at the test results: - **eca8eb7** (Initial commit): PASS - **72d23eb** (set debug off): PASS - **b6aa06f** (enable debug mode): **FAIL** - **bdccc8b** (add startup logging): FAIL The tests passed after the "set debug off" commit but started failing after "enable debug mode". This means the bug was introduced by commit **b6aa06f** (enable debug mode). **Answer: b6aa06f** (or "enable debug mode")
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 362 in → 352 out (342 reasoning)
2abcfcf
JSON Schema Structured Output
(raw) {"commit":"2abcfcf"}
minimax/minimax-m3:high PASS 100% 357 in → 118 out (108 reasoning)
702b9df
minimax/minimax-m3:high__json_schema PASS 100% 307 in → 11 out (10 reasoning)
0e225a0
JSON Schema Structured Output
(raw) {"commit": "0e225a0"}
minimax/minimax-m3:low PASS 100% 361 in → 191 out (164 reasoning)
0add181
minimax/minimax-m3:low__json_schema PASS 100% 350 in → 12 out (0 reasoning)
f7643fb
JSON Schema Structured Output
(raw) { "commit": "f7643fb" }
minimax/minimax-m3:medium PASS 100% 359 in → 190 out (137 reasoning)
**2094c42** (enable debug mode) This is the first commit where the build transitioned from PASS to FAIL, making it the commit that introduced the bug. The subsequent commit (31e8771) inherited the broken state rather than causing it.
minimax/minimax-m3:medium__json_schema PASS 100% 346 in → 13 out (0 reasoning)
c4dccce
JSON Schema Structured Output
(raw) { "commit": "c4dccce" }
minimax/minimax-m3:xhigh PASS 100% 357 in → 100 out (89 reasoning)
e491e33 (enable debug mode)
minimax/minimax-m3:xhigh__json_schema PASS 100% 348 in → 10 out (0 reasoning)
6b57520
JSON Schema Structured Output
(raw) {"commit": "6b57520"}
mistralai/devstral-2512 PASS 100% 211 in → 68 out
Based on the test results, the build started failing at commit `6a8c60c enable debug mode` and continued to fail in the subsequent commit `8bcc8aa add startup logging`. The bad commit that introduced the bug is: `6a8c60c` (or "enable debug mode")
mistralai/devstral-2512__json_schema PASS 100% 215 in → 13 out
a324f10
JSON Schema Structured Output
(raw) {"commit": "a324f10"}
mistralai/mistral-medium-3-5:high PASS 100% 227 in → 505 out (362 reasoning)
The first failing commit in the test results is `4349101 enable debug mode`, as all commits before it passed and all after it failed. This indicates that this commit introduced the bug. 4349101
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 211 in → 328 out (263 reasoning)
e108e70 enable debug mode
JSON Schema Structured Output
(raw) {"commit": "e108e70 enable debug mode"}
mistralai/mistral-medium-3-5:low PASS 100% 219 in → 362 out (289 reasoning)
61ec9ef
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 217 in → 400 out (301 reasoning)
9548245
JSON Schema Structured Output
(raw) {"commit": "9548245"}
mistralai/mistral-medium-3-5:medium PASS 100% 223 in → 543 out (492 reasoning)
bed1904
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 215 in → 409 out (317 reasoning)
60e9678
JSON Schema Structured Output
(raw) {"commit": "60e9678"}
mistralai/mistral-medium-3-5:none PASS 100% 221 in → 146 out (0 reasoning)
The bad commit is **1c59cb4 enable debug mode**. Here’s the reasoning: 1. The test results show the build was working up to commit **5befbf7 set debug off** (PASS). 2. The first failing test is at **1c59cb4 enable debug mode** (FAIL). 3. The next commit **8f5937c add startup logging** also fails, but since the bug was already introduced in **1c59cb4**, that is the culprit. Thus, **1c59cb4** is the commit that broke the build. Answer: `1c59cb4` or `enable debug mode`.
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 211 in → 12 out (0 reasoning)
f7643fb
JSON Schema Structured Output
(raw) {"commit": "f7643fb"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 225 in → 510 out (402 reasoning)
d1d0458
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 211 in → 308 out (236 reasoning)
77fee5b
JSON Schema Structured Output
(raw) {"commit": "77fee5b"}
moonshotai/kimi-k2.6:high PASS 100% 187 in → 326 out (322 reasoning)
74e96fb
moonshotai/kimi-k2.6:high__json_schema PASS 100% 187 in → 212 out (219 reasoning)
05aad72
JSON Schema Structured Output
(raw) { "commit": "05aad72" }
moonshotai/kimi-k2.6:low PASS 100% 189 in → 341 out (322 reasoning)
473073f
moonshotai/kimi-k2.6:low__json_schema PASS 100% 379 in → 372 out (363 reasoning)
723d146
JSON Schema Structured Output
(raw) {"commit":"723d146"}
moonshotai/kimi-k2.6:medium PASS 100% 191 in → 683 out (688 reasoning)
b9177db
moonshotai/kimi-k2.6:xhigh PASS 100% 191 in → 310 out (307 reasoning)
71cc152
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 203 in → 397 out (384 reasoning)
3ad3abd
JSON Schema Structured Output
(raw) { "commit": "3ad3abd" }
moonshotai/kimi-k2.7-code:high PASS 100% 188 in → 92 out (85 reasoning)
9f8c614
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 195 in → 146 out (132 reasoning)
c26c2ca
JSON Schema Structured Output
(raw) { "commit": "c26c2ca" }
moonshotai/kimi-k2.7-code:low PASS 100% 193 in → 105 out (95 reasoning)
c4b1f35
moonshotai/kimi-k2.7-code:medium PASS 100% 185 in → 147 out (142 reasoning)
521cf61
moonshotai/kimi-k2.7-code:xhigh PASS 100% 193 in → 301 out (289 reasoning)
7c0dff5 (enable debug mode)
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 189 in → 109 out (95 reasoning)
e91f4ce
JSON Schema Structured Output
(raw) { "commit": "e91f4ce" }
moonshotai/kimi-k3:high PASS 100% 266 in → 208 out (122 reasoning)
The bad commit is **70cde84** ("enable debug mode"). Reasoning: `29cab13` and `d47b963` both pass, while `70cde84` is the first commit to fail. The later commit `f3642d1` also fails only because it inherits the breakage introduced in `70cde84`.
moonshotai/kimi-k3:high__json_schema PASS 100% 374 in → 135 out (111 reasoning)
24ce4f5
JSON Schema Structured Output
(raw) {"commit":"24ce4f5"}
moonshotai/kimi-k3:low PASS 100% 270 in → 316 out (126 reasoning)
**Bad commit: `e443762` — "enable debug mode"** **Reasoning:** Using a bisect-style analysis of the test results: | Commit | Result | |--------|--------| | 86947a6 Initial commit | ✅ PASS | | f7d710a set debug off | ✅ PASS | | **e443762 enable debug mode** | ❌ **FAIL** ← first failure | | 3ca5bc5 add startup logging | ❌ FAIL | The last passing commit is `f7d710a`, and the first failing commit is `e443762`. Since the build broke exactly when "enable debug mode" was introduced and remained broken afterward, `e443762` is the commit that introduced the bug. The subsequent commit (`3ca5bc5`) merely inherited the broken state — it didn't cause it.
moonshotai/kimi-k3:low__json_schema PASS 100% 278 in → 285 out (108 reasoning)
**Bad commit: `0d9a245`** — "enable debug mode" **Reasoning:** - `6ba034d` (Initial commit): PASS ✓ - `a9eb63e` (set debug off): PASS ✓ - `0d9a245` (enable debug mode): **FAIL ✗** ← first failing commit - `1f6f4b4` (add startup logging): FAIL ✗ Since `a9eb63e` passes and `0d9a245` is the first commit where the build breaks, commit **`0d9a245`** ("enable debug mode") introduced the bug. The subsequent commit `1f6f4b4` still fails because it builds on top of the broken commit.
moonshotai/kimi-k3:max PASS 100% 268 in → 387 out (172 reasoning)
## Bad Commit: `a74c8a7` ("enable debug mode") **Reasoning:** Walking through the commits chronologically: | Commit | Subject | Result | |--------|---------|--------| | 7ead79f | Initial commit | ✅ PASS | | 265241f | set debug off | ✅ PASS | | **a74c8a7** | **enable debug mode** | ❌ **FAIL** | | 0703264 | add startup logging | ❌ FAIL | The build passed at `265241f` but broke at the very next commit, `a74c8a7`. Since `0703264` also fails but its parent (`a74c8a7`) already fails, `0703264` merely inherited the broken state. **The bug was introduced in `a74c8a7` ("enable debug mode")** — likely enabling debug mode triggered a crash, missing config, or an assertion failure at runtime.
moonshotai/kimi-k3:max__json_schema PASS 100% 272 in → 268 out (94 reasoning)
**Bad commit: `2fa9a18` — "enable debug mode"** **Reasoning:** - `5f61e17` (Initial commit): PASS - `39d83ea` (set debug off): PASS — last known good commit - `2fa9a18` (enable debug mode): FAIL — first commit where the build breaks - `ee158d2` (add startup logging): FAIL — still broken, but it inherited the bug Since `39d83ea` passes and its immediate child `2fa9a18` fails, `2fa9a18` ("enable debug mode") is the commit that introduced the bug. The subsequent commit `ee158d2` fails only because the bug was already present.
moonshotai/kimi-k3:medium PASS 100% 272 in → 87 out (63 reasoning)
200afff — “enable debug mode”
moonshotai/kimi-k3:medium__json_schema PASS 100% 272 in → 238 out (219 reasoning)
f875f32
moonshotai/kimi-k3:xhigh PASS 100% 266 in → 340 out (176 reasoning)
**Bad commit: `fe85d92` — "enable debug mode"** **Reasoning:** - `5e44821` (Initial commit): PASS - `868cfde` (set debug off): PASS - `fe85d92` (enable debug mode): **FAIL** ← first failure - `0710ad5` (add startup logging): FAIL Since `868cfde` is the last passing commit and `fe85d92` is the very next commit where the build breaks, the bug was introduced in **`fe85d92` ("enable debug mode")**. The subsequent commit `0710ad5` still fails, which confirms it inherited the broken state rather than introducing it.
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 266 in → 297 out (140 reasoning)
**Bad commit: `c910813` — "enable debug mode"** **Reasoning:** - `9ac8725` (Initial commit): PASS - `9e587be` (set debug off): PASS ← last known good commit - `c910813` (enable debug mode): **FAIL** ← first bad commit - `23d6215` (add startup logging): FAIL Since `9e587be` passes and the very next commit `c910813` fails, the bug was introduced by **`c910813`** ("enable debug mode"). The later commit (`23d6215`) still fails because the bug persists, but it didn't introduce it.
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 222 in → 286 out (249 reasoning)
cf6d073
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 220 in → 211 out (209 reasoning)
fafd942
JSON Schema Structured Output
(raw) { "commit": "fafd942" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 228 in → 286 out (271 reasoning)
4a49c0c
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 224 in → 126 out (121 reasoning)
enable debug mode
JSON Schema Structured Output
(raw) { "commit" : "enable debug mode" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 224 in → 354 out (293 reasoning)
f8114ea (or "enable debug mode")
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 228 in → 227 out (187 reasoning)
45537d5
JSON Schema Structured Output
(raw) {"commit": "45537d5" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 220 in → 44 out (0 reasoning)
**Bad commit:** `384aa59` – “enable debug mode” (This commit introduced the exit‑code 1 failure that first appears at the test run for that commit.)
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 224 in → 15 out (0 reasoning)
366d10a
JSON Schema Structured Output
(raw) { "commit": "366d10a" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 220 in → 304 out (296 reasoning)
fd954dc
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 224 in → 391 out (381 reasoning)
524ebfe
JSON Schema Structured Output
(raw) { "commit": "524ebfe" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 222 in → 174 out (149 reasoning)
6c33a50
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 228 in → 203 out (161 reasoning)
609d9f8
JSON Schema Structured Output
(raw) { "commit": "609d9f8" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 218 in → 151 out (128 reasoning)
e11dce4
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 228 in → 225 out (172 reasoning)
03ea903
JSON Schema Structured Output
(raw) { "commit": "03ea903" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 222 in → 188 out (165 reasoning)
525b0e8
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 220 in → 290 out (239 reasoning)
b4ce435
JSON Schema Structured Output
(raw) { "commit": "b4ce435" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 228 in → 101 out (0 reasoning)
The test results show that the build passes on commits `5e900ba` and `e109171`, but fails on `f73d366` and `9492e25`. Since `f73d366 enable debug mode` is the first commit in the history (from oldest to newest) where the test fails, it is the bad commit that introduced the bug. Answer: f73d366
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 222 in → 16 out (0 reasoning)
6185498
JSON Schema Structured Output
(raw) { "commit": "6185498" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 224 in → 174 out (147 reasoning)
c7d9335 (enable debug mode)
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 228 in → 158 out (119 reasoning)
d15ac88
JSON Schema Structured Output
(raw) { "commit": "d15ac88" }
openai/gpt-5.4-mini:high PASS 100% 197 in → 85 out (75 reasoning)
14587b3
openai/gpt-5.4-mini:high__json_schema PASS 100% 236 in → 97 out (77 reasoning)
cdd4fd8
JSON Schema Structured Output
(raw) {"commit":"cdd4fd8"}
openai/gpt-5.4-mini:low PASS 100% 191 in → 34 out (24 reasoning)
7ca447e
openai/gpt-5.4-mini:low__json_schema PASS 100% 228 in → 33 out (14 reasoning)
d38f566
JSON Schema Structured Output
(raw) {"commit":"d38f566"}
openai/gpt-5.4-mini:medium PASS 100% 183 in → 64 out (55 reasoning)
413333a
openai/gpt-5.4-mini:medium__json_schema PASS 100% 232 in → 74 out (54 reasoning)
4a7be18
JSON Schema Structured Output
(raw) {"commit":"4a7be18"}
openai/gpt-5.4-mini:none PASS 100% 197 in → 9 out (0 reasoning)
11ad6b0
openai/gpt-5.4-mini:none__json_schema PASS 100% 238 in → 19 out (0 reasoning)
62e8c0a
JSON Schema Structured Output
(raw) {"commit":"62e8c0a"}
openai/gpt-5.4-mini:xhigh PASS 100% 183 in → 163 out (154 reasoning)
301a440
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 236 in → 168 out (149 reasoning)
7ab4eff
JSON Schema Structured Output
(raw) {"commit":"7ab4eff"}
openai/gpt-5.4-nano:high PASS 100% 189 in → 81 out (71 reasoning)
7542b44
openai/gpt-5.4-nano:high__json_schema PASS 100% 230 in → 86 out (68 reasoning)
054c107
JSON Schema Structured Output
(raw) {"commit":"054c107"}
openai/gpt-5.4-nano:low PASS 100% 197 in → 33 out (22 reasoning)
5f8382f
openai/gpt-5.4-nano:low__json_schema PASS 100% 236 in → 46 out (26 reasoning)
2c1333a
JSON Schema Structured Output
(raw) {"commit":"2c1333a"}
openai/gpt-5.4-nano:medium PASS 100% 191 in → 43 out (33 reasoning)
be9594e
openai/gpt-5.4-nano:medium__json_schema PASS 100% 228 in → 41 out (22 reasoning)
82956a7
JSON Schema Structured Output
(raw) {"commit":"82956a7"}
openai/gpt-5.4-nano:none PASS 100% 189 in → 10 out (0 reasoning)
196163e enable debug mode
openai/gpt-5.4-nano:none__json_schema PASS 100% 230 in → 16 out (0 reasoning)
fea0459
JSON Schema Structured Output
(raw) {"commit":"fea0459"}
openai/gpt-5.4-nano:xhigh PASS 100% 191 in → 93 out (83 reasoning)
61103f2
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 234 in → 154 out (135 reasoning)
e964d92
JSON Schema Structured Output
(raw) {"commit":"e964d92"}
openai/gpt-5.4:high PASS 100% 193 in → 51 out (39 reasoning)
4c9cf3d
openai/gpt-5.4:high__json_schema PASS 100% 232 in → 67 out (48 reasoning)
eef1de9
JSON Schema Structured Output
(raw) {"commit":"eef1de9"}
openai/gpt-5.4:low PASS 100% 193 in → 27 out (17 reasoning)
c387e51
openai/gpt-5.4:low__json_schema PASS 100% 228 in → 37 out (19 reasoning)
153285a
JSON Schema Structured Output
(raw) {"commit":"153285a"}
openai/gpt-5.4:medium PASS 100% 193 in → 54 out (44 reasoning)
d094ea7
openai/gpt-5.4:medium__json_schema PASS 100% 234 in → 54 out (35 reasoning)
be6cdbe
JSON Schema Structured Output
(raw) {"commit":"be6cdbe"}
openai/gpt-5.4:none PASS 100% 193 in → 9 out (0 reasoning)
224c4f2
openai/gpt-5.4:none__json_schema PASS 100% 232 in → 16 out (0 reasoning)
6043027
JSON Schema Structured Output
(raw) {"commit":"6043027"}
openai/gpt-5.4:xhigh PASS 100% 189 in → 79 out (69 reasoning)
b8256cc
openai/gpt-5.4:xhigh__json_schema PASS 100% 234 in → 217 out (196 reasoning)
b6b42b4
JSON Schema Structured Output
(raw) {"commit":"b6b42b4"}
openai/gpt-5.5:high PASS 100% 187 in → 44 out (34 reasoning)
62029c6
openai/gpt-5.5:high__json_schema PASS 100% 230 in → 16 out (0 reasoning)
1718875
JSON Schema Structured Output
(raw) {"commit":"1718875"}
openai/gpt-5.5:low PASS 100% 195 in → 32 out (22 reasoning)
2efdc44
openai/gpt-5.5:low__json_schema PASS 100% 232 in → 16 out (0 reasoning)
211ab53
JSON Schema Structured Output
(raw) {"commit":"211ab53"}
openai/gpt-5.5:medium PASS 100% 193 in → 32 out (22 reasoning)
dad87d4
openai/gpt-5.5:medium__json_schema PASS 100% 236 in → 18 out (0 reasoning)
e345e2c
JSON Schema Structured Output
(raw) {"commit":"e345e2c"}
openai/gpt-5.5:none PASS 100% 193 in → 8 out (0 reasoning)
3df8298
openai/gpt-5.5:none__json_schema PASS 100% 232 in → 18 out (0 reasoning)
e0bc56b
JSON Schema Structured Output
(raw) {"commit":"e0bc56b"}
openai/gpt-5.5:xhigh PASS 100% 197 in → 55 out (45 reasoning)
e49285f
openai/gpt-5.5:xhigh__json_schema PASS 100% 226 in → 104 out (85 reasoning)
e203bc4
JSON Schema Structured Output
(raw) {"commit":"e203bc4"}
openai/gpt-5.6-luna:high PASS 100% 189 in → 8 out (0 reasoning)
73b979d
openai/gpt-5.6-luna:high__json_schema PASS 100% 238 in → 17 out (0 reasoning)
7de0352
JSON Schema Structured Output
(raw) {"commit":"7de0352"}
openai/gpt-5.6-luna:low PASS 100% 193 in → 9 out (0 reasoning)
1e3272e
openai/gpt-5.6-luna:low__json_schema PASS 100% 236 in → 18 out (0 reasoning)
cb11e9c
JSON Schema Structured Output
(raw) {"commit":"cb11e9c"}
openai/gpt-5.6-luna:max PASS 100% 193 in → 9 out (0 reasoning)
09e8be6
openai/gpt-5.6-luna:max__json_schema PASS 100% 240 in → 17 out (0 reasoning)
7813ad9
JSON Schema Structured Output
(raw) {"commit":"7813ad9"}
openai/gpt-5.6-luna:medium PASS 100% 195 in → 9 out (0 reasoning)
4f5251d
openai/gpt-5.6-luna:medium__json_schema PASS 100% 230 in → 17 out (0 reasoning)
5658abf
JSON Schema Structured Output
(raw) {"commit":"5658abf"}
openai/gpt-5.6-luna:none PASS 100% 195 in → 8 out (0 reasoning)
56a3622
openai/gpt-5.6-luna:none__json_schema PASS 100% 238 in → 18 out (0 reasoning)
f6eb57c
JSON Schema Structured Output
(raw) {"commit":"f6eb57c"}
openai/gpt-5.6-luna:xhigh PASS 100% 189 in → 7 out (0 reasoning)
0414279
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 228 in → 16 out (0 reasoning)
357c359
JSON Schema Structured Output
(raw) {"commit":"357c359"}
openai/gpt-5.6-sol:high PASS 100% 187 in → 8 out (0 reasoning)
c17c536
openai/gpt-5.6-sol:high__json_schema PASS 100% 234 in → 18 out (0 reasoning)
0f51ad9
JSON Schema Structured Output
(raw) {"commit":"0f51ad9"}
openai/gpt-5.6-sol:low PASS 100% 193 in → 11 out (0 reasoning)
2a9a9d4
openai/gpt-5.6-sol:low__json_schema PASS 100% 242 in → 19 out (0 reasoning)
91f2f9a
JSON Schema Structured Output
(raw) {"commit":"91f2f9a"}
openai/gpt-5.6-sol:max PASS 100% 199 in → 8 out (0 reasoning)
d29c912
openai/gpt-5.6-sol:max__json_schema PASS 100% 238 in → 17 out (0 reasoning)
7dd6412
JSON Schema Structured Output
(raw) {"commit":"7dd6412"}
openai/gpt-5.6-sol:medium PASS 100% 193 in → 8 out (0 reasoning)
dcb8934
openai/gpt-5.6-sol:medium__json_schema PASS 100% 228 in → 17 out (0 reasoning)
29d3779
JSON Schema Structured Output
(raw) {"commit":"29d3779"}
openai/gpt-5.6-sol:none PASS 100% 195 in → 10 out (0 reasoning)
c13e1e0
openai/gpt-5.6-sol:none__json_schema PASS 100% 238 in → 17 out (0 reasoning)
4dbcdb7
JSON Schema Structured Output
(raw) {"commit":"4dbcdb7"}
openai/gpt-5.6-sol:xhigh PASS 100% 197 in → 10 out (0 reasoning)
f8f1a98
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 234 in → 17 out (0 reasoning)
10d563a
JSON Schema Structured Output
(raw) {"commit":"10d563a"}
openai/gpt-5.6-terra:high PASS 100% 193 in → 8 out (0 reasoning)
175f84d
openai/gpt-5.6-terra:high__json_schema PASS 100% 230 in → 16 out (0 reasoning)
473a893
JSON Schema Structured Output
(raw) {"commit":"473a893"}
openai/gpt-5.6-terra:low PASS 100% 189 in → 11 out (0 reasoning)
17e9778 enable debug mode
openai/gpt-5.6-terra:low__json_schema PASS 100% 234 in → 17 out (0 reasoning)
e549c36
JSON Schema Structured Output
(raw) {"commit":"e549c36"}
openai/gpt-5.6-terra:max PASS 100% 193 in → 12 out (0 reasoning)
0e7fd41 enable debug mode
openai/gpt-5.6-terra:max__json_schema PASS 100% 230 in → 91 out (72 reasoning)
a003c25
JSON Schema Structured Output
(raw) {"commit":"a003c25"}
openai/gpt-5.6-terra:medium PASS 100% 193 in → 8 out (0 reasoning)
76652a4
openai/gpt-5.6-terra:medium__json_schema PASS 100% 236 in → 18 out (0 reasoning)
6b0fe89
JSON Schema Structured Output
(raw) {"commit":"6b0fe89"}
openai/gpt-5.6-terra:none PASS 100% 193 in → 8 out (0 reasoning)
e9262cd
openai/gpt-5.6-terra:none__json_schema PASS 100% 228 in → 18 out (0 reasoning)
e8725d8
JSON Schema Structured Output
(raw) {"commit":"e8725d8"}
openai/gpt-5.6-terra:xhigh PASS 100% 189 in → 8 out (0 reasoning)
aca1f59
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 234 in → 46 out (27 reasoning)
4586c29
JSON Schema Structured Output
(raw) {"commit":"4586c29"}
openai/gpt-oss-120b:high PASS 100% 256 in → 487 out (468 reasoning)
5e7bef5 enable debug mode
openai/gpt-oss-120b:low PASS 100% 241 in → 49 out (26 reasoning)
64c66ea
openai/gpt-oss-120b:low__json_schema PASS 100% 247 in → 65 out (44 reasoning)
7b513d1
JSON Schema Structured Output
(raw) { "commit": "7b513d1" }
openai/gpt-oss-120b:medium PASS 100% 256 in → 110 out (90 reasoning)
4cf0742 (enable debug mode)
openai/gpt-oss-120b:medium__json_schema PASS 100% 242 in → 103 out (91 reasoning)
88f084d
JSON Schema Structured Output
(raw) { "commit": "88f084d" }
openai/gpt-oss-120b:xhigh PASS 100% 252 in → 454 out (436 reasoning)
5029be3 (enable debug mode)
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 254 in → 335 out (310 reasoning)
bb4eb6a
JSON Schema Structured Output
(raw) { "commit": "bb4eb6a" }
openai/gpt-oss-20b:high PASS 100% 243 in → 756 out (741 reasoning)
The buggy commit is **86c7967** – the “enable debug mode” commit.
openai/gpt-oss-20b:high__json_schema PASS 100% 239 in → 497 out (460 reasoning)
e9cc281
JSON Schema Structured Output
(raw) {"commit":"e9cc281"}
openai/gpt-oss-20b:low PASS 100% 239 in → 101 out (61 reasoning)
The bad commit is **9c823c5 (subject: “enable debug mode”)**.
openai/gpt-oss-20b:medium PASS 100% 252 in → 463 out (460 reasoning)
enable debug mode
openai/gpt-oss-20b:medium__json_schema PASS 100% 241 in → 200 out (171 reasoning)
9b2c38e
JSON Schema Structured Output
(raw) {"commit":"9b2c38e"}
openai/gpt-oss-20b:xhigh PASS 100% 246 in → 402 out (370 reasoning)
046970e (subject: “enable debug mode”)
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 250 in → 795 out (784 reasoning)
495610b
JSON Schema Structured Output
(raw) { "commit": "495610b" }
poolside/laguna-m.1:high PASS 100% 228 in → 592 out (531 reasoning)
The bad commit that introduced the build failure is **926513a** ("enable debug mode"). This is the first commit in the test results that caused the tests to fail, following a series of passing builds. Enabling debug mode likely introduced an issue that broke the build.
poolside/laguna-m.1:high__json_schema PASS 100% 224 in → 700 out (684 reasoning)
d85acd5
JSON Schema Structured Output
(raw) { "commit": "d85acd5" }
poolside/laguna-m.1:low PASS 100% 226 in → 670 out (660 reasoning)
9d77d75
poolside/laguna-m.1:low__json_schema PASS 100% 230 in → 464 out (450 reasoning)
b6b5c76
JSON Schema Structured Output
(raw) {"commit": "b6b5c76"}
poolside/laguna-m.1:medium PASS 100% 228 in → 795 out (725 reasoning)
The bad commit that introduced the bug is **f2e08c3**, as it is the first commit in the history (from oldest to newest) that caused the tests to fail. The commit subject line "enable debug mode" is the one to blame. **Answer:** f2e08c3
poolside/laguna-m.1:medium__json_schema PASS 100% 220 in → 505 out (492 reasoning)
ef9a56a
JSON Schema Structured Output
(raw) {"commit": "ef9a56a"}
poolside/laguna-m.1:none PASS 100% 226 in → 177 out (0 reasoning)
Looking at the git commit history and test results, I can identify the bad commit by analyzing when the tests started failing. From the test results (oldest to newest): - 2c61653 (Initial commit): PASS - 0efe0e5 (set debug off): PASS - 9763e52 (enable debug mode): FAIL - 1066e31 (add startup logging): FAIL The tests passed up to commit 0efe0e5, but failed starting with commit 9763e52. This indicates that commit 9763e52 "enable debug mode" is the bad commit that broke the build, as it's the first commit where the tests began failing. The answer is: 9763e52
poolside/laguna-m.1:none__json_schema PASS 100% 218 in → 12 out (0 reasoning)
fe01d63
JSON Schema Structured Output
(raw) {"commit": "fe01d63"}
poolside/laguna-m.1:xhigh PASS 100% 216 in → 503 out (411 reasoning)
The bad commit is `f70cdf5` with the subject line "enable debug mode". This commit introduced the bug since the tests passed before it (commits `5fbda57` and `9cb1eed`) but failed after its inclusion. The subsequent commit (`4d48d79`) also failed, confirming that the issue originated in `f70cdf5`. Answer: f70cdf5
poolside/laguna-m.1:xhigh__json_schema PASS 100% 222 in → 651 out (633 reasoning)
667159f
JSON Schema Structured Output
(raw) { "commit": "667159f" }
poolside/laguna-xs-2.1:high PASS 100% 222 in → 561 out (448 reasoning)
The bad commit is **7d6cac8** ("enable debug mode"). **Explanation:** - The earliest failure occurs at commit `7d6cac8`, which introduced a change that broke the build. - Subsequent commits (e.g., `da3d8d6`) inherit the failure, confirming the bug was introduced in `7d6cac8`. - The commit subject is "enable debug mode". Answer: **7d6cac8** (or "enable debug mode")
poolside/laguna-xs-2.1:high__json_schema PASS 100% 222 in → 773 out (760 reasoning)
d89fe06
JSON Schema Structured Output
(raw) {"commit": "d89fe06"}
poolside/laguna-xs-2.1:low PASS 100% 218 in → 504 out (440 reasoning)
The bad commit is **d1dbacb** (enable debug mode). The test results show that the build passed before this commit and failed starting from it, making it the first commit to introduce the bug. Subsequent commits (like b7eb0b7) inherit the failure but are not the root cause.
poolside/laguna-xs-2.1:low__json_schema PASS 100% 224 in → 667 out (649 reasoning)
7546fa3
JSON Schema Structured Output
(raw) { "commit": "7546fa3" }
poolside/laguna-xs-2.1:medium PASS 100% 230 in → 324 out (206 reasoning)
The bad commit is c54d53e. **Reasoning:** The test results show that the first failure occurs at commit `c54d53e` ("enable debug mode"). The prior commit `e6b0292` ("set debug off") passed all tests, so the bug was introduced in `c54d53e`. Subsequent commits like `a086b15` inherit the broken state but are not the root cause. **Answer:** c54d53e
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 222 in → 438 out (424 reasoning)
807d7b0
JSON Schema Structured Output
(raw) {"commit": "807d7b0"}
poolside/laguna-xs-2.1:none PASS 100% 226 in → 7 out (0 reasoning)
d7591b1
poolside/laguna-xs-2.1:none__json_schema PASS 100% 222 in → 17 out (0 reasoning)
c836f2b
JSON Schema Structured Output
(raw) { "commit": "c836f2b" }
poolside/laguna-xs-2.1:xhigh PASS 100% 224 in → 725 out (580 reasoning)
The bad commit is identified by the first test failure in the sequence. The test results show: 1. **1c2a9e5 (Initial commit)**: PASS 2. **a23de42 (set debug off)**: PASS 3. **696ea5a (enable debug mode)**: FAIL 4. **891ae1b (add startup logging)**: FAIL The first failure occurs at **696ea5a**, which means this commit introduced the bug. Subsequent commits (like 891ae1b) inherit the failure but are not the root cause. **Answer**: 696ea5a
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 224 in → 490 out (476 reasoning)
e54b05f
JSON Schema Structured Output
(raw) {"commit": "e54b05f"}
poolside/laguna-xs.2:high PASS 100% 265 in → 360 out (246 reasoning)
Based on the test results, I can see that: - Commit 6e59362 (Initial commit): PASS - Commit 4097789 (set debug off): PASS - Commit 0b17c3b (enable debug mode): FAIL - Commit 4ad9a04 (add startup logging): FAIL The first commit where the build broke was **0b17c3b** with the subject "enable debug mode". 0b17c3b
poolside/laguna-xs.2:high__json_schema PASS 100% 263 in → 265 out (248 reasoning)
a42908e enable debug mode
JSON Schema Structured Output
(raw) {"commit": "a42908e enable debug mode"}
poolside/laguna-xs.2:low PASS 100% 257 in → 406 out (311 reasoning)
Looking at the test results by commit: - efdf0d1 Initial commit: PASS - 1d1a630 set debug off: PASS - 4af127c enable debug mode: FAIL (first failure) - 633ddf7 add startup logging: FAIL The bad commit is **4af127c** which introduced the bug when it failed the test for the first time.
poolside/laguna-xs.2:low__json_schema PASS 100% 265 in → 329 out (315 reasoning)
3b72212
JSON Schema Structured Output
(raw) {"commit": "3b72212"}
poolside/laguna-xs.2:medium PASS 100% 261 in → 446 out (297 reasoning)
Based on the test results, I can see that: 1. Commit `f33cf3c` (Initial commit): PASS 2. Commit `c6de34a` (set debug off): PASS 3. Commit `c20c234` (enable debug mode): FAIL 4. Commit `92ef0e7` (add startup logging): FAIL The build was passing after `c6de34a` but started failing at `c20c234`. This indicates that the commit `c20c234` ("enable debug mode") introduced the bug. **Bad commit: c20c234**
poolside/laguna-xs.2:medium__json_schema PASS 100% 263 in → 366 out (352 reasoning)
278195a
JSON Schema Structured Output
(raw) {"commit": "278195a"}
poolside/laguna-xs.2:none PASS 100% 259 in → 125 out (0 reasoning)
Looking at the test results, I can see that: - Commit `84eadfe` (Initial commit): PASS - Commit `0ce747a` (set debug off): PASS - Commit `8b94328` (enable debug mode): FAIL - Commit `6462d42` (add startup logging): FAIL The build started failing at commit `8b94328` ("enable debug mode"), so this is the bad commit that introduced the bug. **Answer: 8b94328**
poolside/laguna-xs.2:none__json_schema PASS 100% 265 in → 13 out (0 reasoning)
3668f37
JSON Schema Structured Output
(raw) {"commit": "3668f37"}
poolside/laguna-xs.2:xhigh PASS 100% 263 in → 360 out (252 reasoning)
Based on the test results, I can see that: - f74f6df (Initial commit): PASS - 91b9d6f (set debug off): PASS - 9ab0a47 (enable debug mode): FAIL - c4b35d6 (add startup logging): FAIL The commit `9ab0a47` ("enable debug mode") is the first one that introduced the failure. This is the bad commit that broke the build.
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 263 in → 261 out (247 reasoning)
9f19e6a
JSON Schema Structured Output
(raw) {"commit": "9f19e6a"}
qwen/qwen3.6-27b:high PASS 100% 220 in → 432 out (333 reasoning)
16ea01a
qwen/qwen3.6-27b:high__json_schema PASS 100% 224 in → 452 out (339 reasoning)
d0890bc
JSON Schema Structured Output
(raw) { "commit": "d0890bc" }
qwen/qwen3.6-27b:low PASS 100% 224 in → 444 out (343 reasoning)
25b6d25
qwen/qwen3.6-27b:low__json_schema PASS 100% 224 in → 852 out (614 reasoning)
889f286
JSON Schema Structured Output
(raw) { "commit": "889f286" }
qwen/qwen3.6-27b:medium PASS 100% 216 in → 756 out (625 reasoning)
048b5eb
qwen/qwen3.6-27b:medium__json_schema PASS 100% 216 in → 16 out (0 reasoning)
b9a16dd
JSON Schema Structured Output
(raw) { "commit": "b9a16dd" }
qwen/qwen3.6-27b:none PASS 100% 222 in → 7 out (0 reasoning)
3b6eefb
qwen/qwen3.6-27b:none__json_schema PASS 100% 224 in → 18 out (0 reasoning)
e5f9a49
JSON Schema Structured Output
(raw) { "commit": "e5f9a49" }
qwen/qwen3.6-27b:xhigh PASS 100% 220 in → 568 out (442 reasoning)
0afe329
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 222 in → 620 out (505 reasoning)
22300aa
JSON Schema Structured Output
(raw) { "commit": "22300aa" }
qwen/qwen3.6-35b-a3b:high PASS 100% 214 in → 415 out (337 reasoning)
a7eca5f
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 226 in → 463 out (373 reasoning)
340a246
JSON Schema Structured Output
(raw) { "commit": "340a246" }
qwen/qwen3.6-35b-a3b:low PASS 100% 220 in → 418 out (408 reasoning)
4cfba22
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 218 in → 1,248 out (997 reasoning)
692fc7f
JSON Schema Structured Output
(raw) { "commit": "692fc7f" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 218 in → 359 out (295 reasoning)
06bdc7c
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 220 in → 400 out (334 reasoning)
bb19df0
JSON Schema Structured Output
(raw) { "commit": "bb19df0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 214 in → 7 out (0 reasoning)
8f35cd8
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 222 in → 16 out (0 reasoning)
1caf670
JSON Schema Structured Output
(raw) { "commit": "1caf670" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 224 in → 339 out (328 reasoning)
a9fd732
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 222 in → 519 out (386 reasoning)
692041a
JSON Schema Structured Output
(raw) { "commit": "692041a" }
qwen/qwen3.6-flash:high PASS 100% 224 in → 3,144 out (2,968 reasoning)
Based on the commit history and test results, we can determine the "bad" commit by looking for the first commit in the sequence where the test results change from **PASS** to **FAIL**. 1. **34f35c5** (Initial commit): **PASS** 2. **14a2701** (set debug off): **PASS** (Still working) 3. **d6fb636** (enable debug mode): **FAIL** (The bug was introduced here) 4. **28f593b** (add startup logging): **FAIL** (The build remains broken) The first commit to cause the failure is `d6fb636`. **d6fb636** (or "enable debug mode")
qwen/qwen3.6-flash:low PASS 100% 218 in → 516 out (506 reasoning)
343fdcd
qwen/qwen3.6-flash:medium PASS 100% 224 in → 322 out (311 reasoning)
0ca0f7f
qwen/qwen3.6-flash:none PASS 100% 222 in → 7 out (0 reasoning)
6d65f3c
qwen/qwen3.6-flash:xhigh PASS 100% 222 in → 355 out (343 reasoning)
825c11f
qwen/qwen3.7-flash:high PASS 100% 222 in → 618 out (606 reasoning)
35c3942
qwen/qwen3.7-flash:low PASS 100% 222 in → 245 out (233 reasoning)
0143010
qwen/qwen3.7-flash:max PASS 100% 218 in → 726 out (716 reasoning)
3ab01db
qwen/qwen3.7-flash:medium PASS 100% 220 in → 306 out (294 reasoning)
8d9173f
qwen/qwen3.7-flash:none PASS 100% 222 in → 5 out (0 reasoning)
bcba956
qwen/qwen3.7-flash:xhigh PASS 100% 210 in → 450 out (441 reasoning)
3efd8dd
qwen/qwen3.7-max:high PASS 100% 220 in → 334 out (208 reasoning)
The bad commit is **059a362** (`enable debug mode`). **Reasoning:** Looking at the test results chronologically, the build passes for `b90a5ae` and `9dc9bb3`, but first fails at `059a362`. Since `059a362` is the earliest commit where the test fails, it is the commit that introduced the bug. The subsequent commit `b75806d` also fails because it inherits the bug from `059a362`.
qwen/qwen3.7-max:low PASS 100% 224 in → 183 out (171 reasoning)
2024b24
qwen/qwen3.7-max:medium PASS 100% 218 in → 233 out (223 reasoning)
b461abb
qwen/qwen3.7-max:none PASS 100% 220 in → 6 out (0 reasoning)
52b45fb
qwen/qwen3.7-max:xhigh PASS 100% 222 in → 186 out (175 reasoning)
d5ddc96
qwen/qwen3.7-plus:high PASS 100% 222 in → 423 out (207 reasoning)
The bad commit is **`55b76cd`** ("enable debug mode"). **Reasoning:** Looking at the test results in chronological order: | Commit | Subject | Result | |--------|---------|--------| | `851817a` | Initial commit | ✅ PASS | | `773910d` | set debug off | ✅ PASS | | `55b76cd` | enable debug mode | ❌ **FAIL** | | `dbd1b3a` | add startup logging | ❌ FAIL | The first transition from **PASS → FAIL** occurs at commit `55b76cd` ("enable debug mode"). Since the previous commit (`773910d`) passed and `55b76cd` failed, this is the commit that introduced the bug. The subsequent commit `dbd1b3a` continues to fail as well, confirming it inherited the broken state.
qwen/qwen3.7-plus:low PASS 100% 212 in → 403 out (230 reasoning)
## Identifying the Bad Commit Looking at the test results from oldest to newest: | Commit | Subject | Result | |--------|---------|--------| | eecc0eb | Initial commit | ✅ PASS | | 388dfca | set debug off | ✅ PASS | | f9cc79e | enable debug mode | ❌ FAIL | | f90cd29 | add startup logging | ❌ FAIL | The first failure appears between `388dfca` (PASS) and `f9cc79e` (FAIL). **The bad commit is `f9cc79e` ("enable debug mode")** — this is the first commit that broke the build. All subsequent commits also fail because they inherit the bug introduced here.
qwen/qwen3.7-plus:medium PASS 100% 220 in → 372 out (188 reasoning)
The bad commit is **e9e4da3** ("enable debug mode"). Here's the reasoning: 1. **e9b1ef8** `Initial commit` → PASS ✅ 2. **8b5d45a** `set debug off` → PASS ✅ 3. **e9e4da3** `enable debug mode` → **FAIL** ❌ 4. **df28e40** `add startup logging` → FAIL ❌ (inherits the bug from e9e4da3) The test passed at commit `8b5d45a` and first failed at commit `e9e4da3`, meaning the change introduced in `enable debug mode` broke the build. Since enabling debug mode caused an exit code 1, the debug functionality itself is the source of the bug.
qwen/qwen3.7-plus:none PASS 100% 224 in → 6 out (0 reasoning)
7d9df82
qwen/qwen3.7-plus:xhigh PASS 100% 224 in → 318 out (233 reasoning)
The bad commit is **8f97e1d** ("enable debug mode"). The test results show: - `8023052 set debug off`: **PASS** - `8f97e1d enable debug mode`: **FAIL** ← first failure The commit `8f97e1d` is where the build first broke.
tencent/hy3:high PASS 100% 201 in → 553 out (547 reasoning)
20f2cad
tencent/hy3:high__json_schema PASS 100% 191 in → 437 out (406 reasoning)
13a017b
JSON Schema Structured Output
(raw) { "commit": "13a017b" }
tencent/hy3:low PASS 100% 199 in → 386 out (378 reasoning)
f1c4ddc
tencent/hy3:low__json_schema PASS 100% 197 in → 571 out (492 reasoning)
6718a5f
JSON Schema Structured Output
(raw) { "commit": "6718a5f" }
tencent/hy3:medium PASS 100% 195 in → 306 out (300 reasoning)
42c0377
tencent/hy3:medium__json_schema PASS 100% 197 in → 310 out (259 reasoning)
b34a43c
JSON Schema Structured Output
(raw) { "commit": "b34a43c" }
tencent/hy3:none PASS 100% 198 in → 4 out (0 reasoning)
306a672
tencent/hy3:none__json_schema PASS 100% 196 in → 11 out (0 reasoning)
814facd
JSON Schema Structured Output
(raw) { "commit": "814facd" }
tencent/hy3:xhigh PASS 100% 201 in → 538 out (531 reasoning)
0d46c21
tencent/hy3:xhigh__json_schema PASS 100% 205 in → 560 out (504 reasoning)
6290c93
JSON Schema Structured Output
(raw) { "commit": "6290c93" }
thinkingmachines/inkling-small:high PASS 100% 200 in → 129 out (90 reasoning)
The first failing commit is **84169e4** ("enable debug mode").
thinkingmachines/inkling-small:high__json_schema PASS 100% 198 in → 104 out (91 reasoning)
239db96
JSON Schema Structured Output
(raw) {"commit": "239db96"}
thinkingmachines/inkling-small:low PASS 100% 204 in → 24 out (13 reasoning)
f8dd52b
thinkingmachines/inkling-small:low__json_schema PASS 100% 202 in → 34 out (20 reasoning)
611a14a
JSON Schema Structured Output
(raw) {"commit": "611a14a"}
thinkingmachines/inkling-small:max PASS 100% 194 in → 38 out (27 reasoning)
df7e335
thinkingmachines/inkling-small:max__json_schema PASS 100% 196 in → 33 out (20 reasoning)
93acc69
JSON Schema Structured Output
(raw) {"commit": "93acc69"}
thinkingmachines/inkling-small:medium PASS 100% 202 in → 31 out (21 reasoning)
41dd3af
thinkingmachines/inkling-small:medium__json_schema PASS 100% 198 in → 80 out (65 reasoning)
fcb83d5
JSON Schema Structured Output
(raw) {"commit": "fcb83d5"}
thinkingmachines/inkling-small:none PASS 100% 202 in → 8 out (0 reasoning)
3985d3c
thinkingmachines/inkling-small:xhigh PASS 100% 194 in → 112 out (89 reasoning)
54714d8
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 196 in → 118 out (102 reasoning)
555ba9b
JSON Schema Structured Output
(raw) { "commit": "555ba9b" }
thinkingmachines/inkling:high PASS 100% 204 in → 172 out (121 reasoning)
The build first fails at commit `6c9c34c` ("enable debug mode"), so that is the bad commit. **Answer:** `6c9c34c` (or "enable debug mode")
thinkingmachines/inkling:high__json_schema PASS 100% 202 in → 127 out (111 reasoning)
c7c7e91
JSON Schema Structured Output
(raw) {"commit": "c7c7e91"}
thinkingmachines/inkling:low PASS 100% 198 in → 21 out (10 reasoning)
d32910e
thinkingmachines/inkling:max PASS 100% 200 in → 169 out (120 reasoning)
11b5e16
thinkingmachines/inkling:medium PASS 100% 200 in → 127 out (98 reasoning)
a2c786f
thinkingmachines/inkling:medium__json_schema PASS 100% 198 in → 56 out (39 reasoning)
1a1be31
JSON Schema Structured Output
(raw) { "commit": "1a1be31" }
thinkingmachines/inkling:none PASS 100% 206 in → 9 out (0 reasoning)
d4d371a
thinkingmachines/inkling:none__json_schema PASS 100% 200 in → 11 out (0 reasoning)
c36cc1c
JSON Schema Structured Output
(raw) {"commit":"c36cc1c"}
thinkingmachines/inkling:xhigh PASS 100% 198 in → 120 out (109 reasoning)
eab69df
x-ai/grok-4.3:high PASS 100% 368 in → 582 out (543 reasoning)
ce05c19 The tests pass on 969b26a but first fail on ce05c19, so that commit introduced the bug (subsequent commits remain failing, as expected).
x-ai/grok-4.3:high__json_schema PASS 100% 431 in → 635 out (626 reasoning)
d8238e0
JSON Schema Structured Output
(raw) {"commit":"d8238e0"}
x-ai/grok-4.3:low PASS 100% 382 in → 420 out (337 reasoning)
47b34e9 The test results show that all commits up through bd4b7ed still exit with code 0 (PASS). The very next commit (47b34e9) is the first one that exits with code 1 (FAIL); the newest commit (e09a2e2) also fails. Therefore 47b34e9 is the single commit that introduced the regression.
x-ai/grok-4.3:low__json_schema PASS 100% 433 in → 368 out (358 reasoning)
d60d21d
JSON Schema Structured Output
(raw) {"commit": "d60d21d"}
x-ai/grok-4.3:max PASS 100% 368 in → 443 out (439 reasoning)
2429b95
x-ai/grok-4.3:max__json_schema PASS 100% 433 in → 544 out (535 reasoning)
9cf54d9
JSON Schema Structured Output
(raw) {"commit":"9cf54d9"}
x-ai/grok-4.3:medium PASS 100% 376 in → 501 out (400 reasoning)
2148d45 The test results show that commits 86116e3 and 140a28b both exit with code 0 (PASS). The very next commit in the ancestry, 2148d45, is the first one that exits with code 1 (FAIL). The subsequent commit 742f0b3 also fails, but that is expected once the repository already contains the breaking change. Consequently the regression was introduced by the diff that 2148d45 itself brought in.
x-ai/grok-4.3:medium__json_schema PASS 100% 437 in → 452 out (443 reasoning)
2a379e3
JSON Schema Structured Output
(raw) {"commit":"2a379e3"}
x-ai/grok-4.3:none PASS 100% 370 in → 4 out (0 reasoning)
a75dea2
x-ai/grok-4.3:none__json_schema PASS 100% 429 in → 9 out (0 reasoning)
2b57b81
JSON Schema Structured Output
(raw) {"commit":"2b57b81"}
x-ai/grok-4.3:xhigh PASS 100% 380 in → 437 out (432 reasoning)
98ea4f1
x-ai/grok-4.3:xhigh__json_schema PASS 100% 423 in → 584 out (577 reasoning)
cc57041
JSON Schema Structured Output
(raw) {"commit":"cc57041"}
x-ai/grok-4.5:high PASS 100% 390 in → 248 out (243 reasoning)
eed3c6f
x-ai/grok-4.5:high__json_schema PASS 100% 465 in → 268 out (256 reasoning)
ef7cbda
JSON Schema Structured Output
(raw) { "commit": "ef7cbda" }
x-ai/grok-4.5:low PASS 100% 394 in → 137 out (132 reasoning)
c38a53d
x-ai/grok-4.5:low__json_schema PASS 100% 463 in → 220 out (211 reasoning)
eaa9462
JSON Schema Structured Output
(raw) {"commit":"eaa9462"}
x-ai/grok-4.5:max PASS 100% 394 in → 217 out (212 reasoning)
b4f779a
x-ai/grok-4.5:max__json_schema PASS 100% 463 in → 258 out (247 reasoning)
3d43d3c
JSON Schema Structured Output
(raw) {"commit":"3d43d3c"}
x-ai/grok-4.5:medium PASS 100% 400 in → 300 out (295 reasoning)
50eeb6e
x-ai/grok-4.5:medium__json_schema PASS 100% 463 in → 264 out (256 reasoning)
89666ad
JSON Schema Structured Output
(raw) {"commit":"89666ad"}
x-ai/grok-4.5:xhigh PASS 100% 394 in → 241 out (237 reasoning)
f356c48
x-ai/grok-4.5:xhigh__json_schema PASS 100% 463 in → 270 out (261 reasoning)
aa1af61
JSON Schema Structured Output
(raw) {"commit":"aa1af61"}
z-ai/glm-4.7-flash:high PASS 100% 194 in → 814 out (809 reasoning)
enable debug mode
z-ai/glm-4.7-flash:high__json_schema PASS 100% 194 in → 490 out (417 reasoning)
0f08955
JSON Schema Structured Output
(raw) { "commit": "0f08955" }
z-ai/glm-4.7-flash:low PASS 100% 196 in → 473 out (403 reasoning)
bd75f5d
z-ai/glm-4.7-flash:low__json_schema PASS 100% 198 in → 500 out (415 reasoning)
5d95c9a
JSON Schema Structured Output
(raw) { "commit": "5d95c9a" }
z-ai/glm-4.7-flash:medium PASS 100% 190 in → 735 out (730 reasoning)
enable debug mode
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 194 in → 690 out (551 reasoning)
971bce6
JSON Schema Structured Output
(raw) { "commit": "971bce6" }
z-ai/glm-4.7-flash:none__json_schema PASS 100% 192 in → 13 out (0 reasoning)
2594c95
JSON Schema Structured Output
(raw) { "commit": "2594c95" }
z-ai/glm-4.7-flash:xhigh PASS 100% 194 in → 953 out (946 reasoning)
c28986e
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 194 in → 491 out (409 reasoning)
ebd32bc
JSON Schema Structured Output
(raw) { "commit": "ebd32bc" }
z-ai/glm-5.2:high PASS 100% 199 in → 135 out (129 reasoning)
184423c
z-ai/glm-5.2:high__json_schema PASS 100% 201 in → 170 out (152 reasoning)
453fc85
JSON Schema Structured Output
(raw) {"commit": "453fc85"}
z-ai/glm-5.2:low PASS 100% 199 in → 119 out (112 reasoning)
79cf1e2
z-ai/glm-5.2:low__json_schema PASS 100% 203 in → 155 out (141 reasoning)
d4ddeea
JSON Schema Structured Output
(raw) { "commit": "d4ddeea" }
z-ai/glm-5.2:medium PASS 100% 201 in → 43 out (38 reasoning)
0aaf141
z-ai/glm-5.2:medium__json_schema PASS 100% 205 in → 85 out (70 reasoning)
5b7a3cd
JSON Schema Structured Output
(raw) { "commit": "5b7a3cd" }
z-ai/glm-5.2:none PASS 100% 189 in → 4 out (0 reasoning)
17948be
z-ai/glm-5.2:none__json_schema PASS 100% 195 in → 15 out (0 reasoning)
0768000 enable debug mode
JSON Schema Structured Output
(raw) { "commit": "0768000 enable debug mode" }
z-ai/glm-5.2:xhigh PASS 100% 201 in → 207 out (194 reasoning)
ad4b5ad
z-ai/glm-5.2:xhigh__json_schema PASS 100% 203 in → 259 out (245 reasoning)
c1dde5d
JSON Schema Structured Output
(raw) { "commit": "c1dde5d" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: To identify the **bad commit**, let's analyze the information step-by-step: ### Step 1: Understand the commit order The commits are listed from newest to oldest: - **9f28f9c** – Add startup logging - **5514158** – Enable debug mode - **bfea037** – Set debug off - **1450b9f** – Initial commit ### Step 2: Analyze test results - **1450b9f**: PASS (no issues) - **bfea037**: PASS (no issues) - **5514158**: FAIL (bug introduced here) - **9f28f9c**: FAIL (bug persists here) This means the bug was introduced in commit **5514158** ("enable debug mode"), since it's the first commit where tests fail. ### Step 3: Confirm with commit message The commit message for **5514158** is "enable debug mode". --- **Final Answer:** `5514158 enable debug mode`
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid structured output. Output: 65100e0
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-120b:high__json_schema FAIL 0% 250 in → 13 out (0 reasoning)
3203201bae?
JSON Schema Structured Output
(raw) { "commit": "3203201bae?" }
Failure: Expected target hash/prefix or subject 'enable debug mode'
openai/gpt-oss-20b:low__json_schema FAIL 0% 246 in → 8 out (0 reasoning)
analysis
JSON Schema Structured Output
(raw) {"commit":"analysis"}
Failure: Expected target hash/prefix or subject 'enable debug mode'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0355ba46-a063-96cb-9df8-30815356a0eb","request_id":"0355ba46-a063-96cb-9df8-30815356a0eb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-35e6a638-0af0-97de-a81a-218d0db63641","request_id":"35e6a638-0af0-97de-a81a-218d0db63641"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a35a7992-71f7-9d35-b9df-4605814d213a","request_id":"a35a7992-71f7-9d35-b9df-4605814d213a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9f66183d-e326-9250-84b2-598407ef829d","request_id":"9f66183d-e326-9250-84b2-598407ef829d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d4a2e444-8416-94e9-a00f-45b4dedfec39","request_id":"d4a2e444-8416-94e9-a00f-45b4dedfec39"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-18408718-7592-9f17-96b3-1105513059cb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d6550a87-3e18-914d-a3cb-7e604f346813"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a0449337-a2cc-9b73-a16e-e5f6fcb4991d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b0b0eb30-6e62-9fd5-9d0b-d744d0660737"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4c4fc8f9-5e29-923e-b55f-8f0d8be12f0e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-857506d0-3f61-9ed1-bf9f-9461b66a8443"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f13b6dc5-ad16-9af0-8a1d-4c918fe7ca5d","request_id":"f13b6dc5-ad16-9af0-8a1d-4c918fe7ca5d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-61285242-49ef-959b-a263-1ef441bf8fa0","request_id":"61285242-49ef-959b-a263-1ef441bf8fa0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ddc1f256-da48-9d21-888a-8669fd20fa22","request_id":"ddc1f256-da48-9d21-888a-8669fd20fa22"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-23b2b85e-4fc1-9b22-960a-5e8eb367133c","request_id":"23b2b85e-4fc1-9b22-960a-5e8eb367133c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-981f7a79-b708-94e1-acb6-0ca137f5a014","request_id":"981f7a79-b708-94e1-acb6-0ca137f5a014"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c33d1fc0-fa83-95a0-9246-9222bce7d574","request_id":"c33d1fc0-fa83-95a0-9246-9222bce7d574"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12934b02-506a-9148-bca3-35b6dd673bc7","request_id":"12934b02-506a-9148-bca3-35b6dd673bc7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-09d13cc2-c8a5-9a4a-bd64-a04589f55ecd","request_id":"09d13cc2-c8a5-9a4a-bd64-a04589f55ecd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-767f96a2-226c-9d93-b621-c12138459143","request_id":"767f96a2-226c-9d93-b621-c12138459143"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-530c3cc0-78a2-95bf-adc4-3a217fcef9e3","request_id":"530c3cc0-78a2-95bf-adc4-3a217fcef9e3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: bc49a40{"commit": "Answer: bc49a40"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 21b6e42
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Invalid JSON. Output: 4692f30
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)
Invalid JSON. Output: c00781f
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 190 in → 4 out (0 reasoning)
7dd104d
Failure: Expected target hash/prefix or subject 'enable debug mode'