Linear history with bad commit in the middle - model identifies middle commit as bad
Tests ability to identify a middle commit as bad in a linear history. Evaluates bisect with the bug not at either end.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '#!/bin/bash' > test.sh
  5. 05 echo 'grep "42" config.json' >> test.sh
  6. 06 chmod +x test.sh
  7. 07 echo '{"value": 42}' > config.json
  8. 08 git add test.sh config.json
  9. 09 git commit -m 'Initial commit'
  10. 10 echo '{"value": 42, "name": "default"}' > config.json
  11. 11 git add config.json
  12. 12 git commit -m 'set config value'
  13. 13 echo '{"value": 0}' > config.json
  14. 14 git add config.json
  15. 15 git commit -m 'reset config to zero'
  16. 16 echo '{"value": 100}' > config.json
  17. 17 git add config.json
  18. 18 git commit -m 'final config'
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad. Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
reset config to zero

Scoped model quality, cost, API time, and token usage for git_bisect/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 275 in → 206 out (17 reasoning)
**Bad commit: `d2b5fbf` — "reset config to zero"** Reasoning (bisect logic): - `4bb8928` (Initial commit): PASS ✅ - `91ab55b` (set config value): PASS ✅ - `d2b5fbf` (reset config to zero): FAIL ❌ ← first failing commit - `4e9f8cc` (final config): FAIL ❌ (inherited the breakage) The build transitions from passing to failing at `d2b5fbf`, so that commit introduced the bug.
anthropic/claude-fable-5:high__json_schema PASS 100% 496 in → 17 out (0 reasoning)
dc4b5d2
JSON Schema Structured Output
(raw) {"commit": "dc4b5d2"}
anthropic/claude-fable-5:low PASS 100% 265 in → 24 out (0 reasoning)
First failing commit: **c7242aa** — "reset config to zero"
anthropic/claude-fable-5:low__json_schema PASS 100% 486 in → 14 out (0 reasoning)
4372c66
JSON Schema Structured Output
(raw) {"commit": "4372c66"}
anthropic/claude-fable-5:max PASS 100% 279 in → 440 out (74 reasoning)
## Bad commit: `d0e9277` — "reset config to zero" ### Reasoning Walking through the test results in chronological order (oldest → newest), we look for the first transition from **PASS** to **FAIL**: | Commit | Subject | Result | |---|---|---| | `a9a6ae0` | Initial commit | ✅ PASS | | `e2f5c48` | set config value | ✅ PASS | | `d0e9277` | reset config to zero | ❌ FAIL ← **first failure** | | `75ebd4e` | final config | ❌ FAIL | The build was healthy through `e2f5c48`, then broke at `d0e9277`. All subsequent commits fail because they inherit the breakage — this is exactly the pattern `git bisect` would identify. **Answer: `d0e9277` ("reset config to zero")**
anthropic/claude-fable-5:max__json_schema PASS 100% 484 in → 117 out (43 reasoning)
b2f84d8
JSON Schema Structured Output
(raw) {"commit": "b2f84d8"}
anthropic/claude-fable-5:medium PASS 100% 271 in → 101 out (17 reasoning)
**efafc80** — "reset config to zero" The tests pass through `58b8cca` and first fail at `efafc80`, meaning that commit introduced the bug (likely resetting a config value to zero broke something that depended on the previously set value).
anthropic/claude-fable-5:medium__json_schema PASS 100% 488 in → 16 out (0 reasoning)
44b6d0b
JSON Schema Structured Output
(raw) {"commit": "44b6d0b"}
anthropic/claude-fable-5:xhigh PASS 100% 269 in → 197 out (17 reasoning)
The bad commit is **803e59d** — "reset config to zero". **Reasoning:** - `afaae0d` (Initial commit): PASS - `b4ab811` (set config value): PASS - `803e59d` (reset config to zero): **FAIL** ← first failing commit - `d04ffa0` (final config): FAIL (inherits the breakage) The build passes through `b4ab811` and first fails at `803e59d`, so that commit introduced the bug. All subsequent failures are a consequence of it.
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 488 in → 36 out (19 reasoning)
9c27066
JSON Schema Structured Output
(raw) {"commit": "9c27066"}
anthropic/claude-haiku-4.5:high PASS 100% 242 in → 519 out (298 reasoning)
# Analysis Looking at the test results chronologically: 1. **f6ea3a4** (Initial commit): PASS ✓ 2. **51ed682** (set config value): PASS ✓ 3. **3017cba** (reset config to zero): FAIL ✗ ← First failure 4. **37fc1e3** (final config): FAIL ✗ The build was passing through commit `51ed682` and first failed at commit `3017cba`. This is the commit that introduced the bug. **Answer: `3017cba`** (or "reset config to zero")
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 408 in → 399 out (310 reasoning)
8f8390f
JSON Schema Structured Output
(raw) {"commit": "8f8390f"}
anthropic/claude-haiku-4.5:low PASS 100% 240 in → 419 out (208 reasoning)
# Answer **41e420b** (or "reset config to zero") ## Analysis Looking at the test results in chronological order: - 4e81b44 (Initial commit): **PASS** ✓ - 1378c64 (set config value): **PASS** ✓ - 41e420b (reset config to zero): **FAIL** ✗ - a8b612a (final config): **FAIL** ✗ The first commit where tests fail is **41e420b (reset config to zero)**. This is where the build broke, transitioning from a passing state to a failing state. The subsequent commit (a8b612a) also fails, but it's a consequence of the bad commit that preceded it.
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 412 in → 359 out (274 reasoning)
1934591
JSON Schema Structured Output
(raw) {"commit": "1934591"}
anthropic/claude-haiku-4.5:medium PASS 100% 244 in → 472 out (218 reasoning)
# Answer **0ca53b4** ## Analysis Looking at the test results in chronological order: 1. 9a9330c Initial commit: **PASS** ✓ 2. 0ac3db0 set config value: **PASS** ✓ 3. 0ca53b4 reset config to zero: **FAIL** ✗ ← First failure 4. c951d89 final config: **FAIL** ✗ The bad commit is **0ca53b4 (reset config to zero)** because it's the first commit where the build started failing. The tests passed up through commit 0ac3db0, and began failing at commit 0ca53b4.
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 412 in → 399 out (318 reasoning)
0a66d08
JSON Schema Structured Output
(raw) {"commit": "0a66d08"}
anthropic/claude-haiku-4.5:none PASS 100% 217 in → 159 out (0 reasoning)
# Analysis Looking at the test results in chronological order: 1. **2fd6398** (Initial commit): PASS ✓ 2. **0b45ee2** (set config value): PASS ✓ 3. **2d513b1** (reset config to zero): FAIL ✗ 4. **8bd6e91** (final config): FAIL ✗ The first commit to fail is **2d513b1**, which is where the build broke. The subsequent commit (8bd6e91) also fails, but that's a consequence of the bad commit before it. **Answer: 2d513b1** (or "reset config to zero")
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 378 in → 13 out (0 reasoning)
4cfb24a
JSON Schema Structured Output
(raw) {"commit": "4cfb24a"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 240 in → 367 out (192 reasoning)
# Analysis Looking at the test results in chronological order (oldest first): 1. **a906e73** Initial commit: PASS ✓ 2. **0aa3999** set config value: PASS ✓ 3. **ea25bdf** reset config to zero: **FAIL** ✗ ← First failure 4. **f7584b6** final config: FAIL ✗ The bad commit is **ea25bdf** (or "reset config to zero"), as it's the first commit where the tests started failing.
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 410 in → 375 out (304 reasoning)
478cc90
JSON Schema Structured Output
(raw) {"commit": "478cc90"}
anthropic/claude-opus-4.6:high PASS 100% 211 in → 35 out (0 reasoning)
The bad commit is `5297149` ("reset config to zero"), as it is the first commit where the test changes from PASS to FAIL.
anthropic/claude-opus-4.6:high__json_schema PASS 100% 385 in → 12 out (0 reasoning)
f3374a7
JSON Schema Structured Output
(raw) {"commit":"f3374a7"}
anthropic/claude-opus-4.6:low PASS 100% 209 in → 8 out (0 reasoning)
600b6fa
anthropic/claude-opus-4.6:low__json_schema PASS 100% 383 in → 10 out (0 reasoning)
cb13054
JSON Schema Structured Output
(raw) {"commit":"cb13054"}
anthropic/claude-opus-4.6:max PASS 100% 211 in → 264 out (97 reasoning)
**Bad commit: `7c1c971`** ("reset config to zero") **Reasoning:** The test results show that all commits up to and including `ee0e682` (set config value) **pass**, but `7c1c971` (reset config to zero) is the first commit that **fails**. Since the subsequent commit `2e4cbff` also fails, the bug was introduced at `7c1c971` and carried forward. This is the classic `git bisect` pattern — the bad commit is the first one where the test transitions from PASS → FAIL.
anthropic/claude-opus-4.6:max__json_schema PASS 100% 377 in → 77 out (50 reasoning)
6feb757
JSON Schema Structured Output
(raw) {"commit":"6feb757"}
anthropic/claude-opus-4.6:medium PASS 100% 211 in → 7 out (0 reasoning)
b081d32
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 379 in → 11 out (0 reasoning)
995c34b
JSON Schema Structured Output
(raw) {"commit":"995c34b"}
anthropic/claude-opus-4.6:none PASS 100% 213 in → 36 out (0 reasoning)
The bad commit is **0696f73** ("reset config to zero"), as it is the first commit where the test changes from PASS to FAIL.
anthropic/claude-opus-4.6:none__json_schema PASS 100% 383 in → 12 out (0 reasoning)
4e3cd24
JSON Schema Structured Output
(raw) {"commit":"4e3cd24"}
anthropic/claude-opus-4.6:xhigh PASS 100% 213 in → 39 out (0 reasoning)
The bad commit is **3c4b1f1** ("reset config to zero"), as it is the first commit where the test changes from PASS to FAIL.
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 373 in → 11 out (0 reasoning)
84f6962
JSON Schema Structured Output
(raw) {"commit":"84f6962"}
anthropic/claude-opus-4.7:high PASS 100% 274 in → 8 out (0 reasoning)
c01f521
anthropic/claude-opus-4.7:high__json_schema PASS 100% 493 in → 17 out (0 reasoning)
212cf6b
JSON Schema Structured Output
(raw) {"commit":"212cf6b"}
anthropic/claude-opus-4.7:low PASS 100% 272 in → 10 out (0 reasoning)
18d2c12
anthropic/claude-opus-4.7:low__json_schema PASS 100% 493 in → 17 out (0 reasoning)
bbfe550
JSON Schema Structured Output
(raw) {"commit":"bbfe550"}
anthropic/claude-opus-4.7:max PASS 100% 270 in → 342 out (52 reasoning)
Looking at the test results, I can identify the bad commit by finding where the tests transitioned from PASS to FAIL: - `561788d` Initial commit: **PASS** - `5fd7f9d` set config value: **PASS** - `638af6d` reset config to zero: **FAIL** ← First failure - `cd1e120` final config: FAIL (inherits the bug) The bad commit is **`638af6d`** ("reset config to zero"). This is the commit where the build first started failing, and the subsequent commit `cd1e120` also fails because it inherits the broken state.
anthropic/claude-opus-4.7:max__json_schema PASS 100% 505 in → 17 out (0 reasoning)
322f6b2
JSON Schema Structured Output
(raw) {"commit":"322f6b2"}
anthropic/claude-opus-4.7:medium PASS 100% 276 in → 10 out (0 reasoning)
b22e6f4
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 487 in → 16 out (0 reasoning)
f894d85
JSON Schema Structured Output
(raw) {"commit":"f894d85"}
anthropic/claude-opus-4.7:none PASS 100% 280 in → 9 out (0 reasoning)
a4ede51
anthropic/claude-opus-4.7:none__json_schema PASS 100% 497 in → 16 out (0 reasoning)
0d74192
JSON Schema Structured Output
(raw) {"commit":"0d74192"}
anthropic/claude-opus-4.7:xhigh PASS 100% 268 in → 9 out (0 reasoning)
33f284d
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 497 in → 17 out (0 reasoning)
7d97c01
JSON Schema Structured Output
(raw) {"commit":"7d97c01"}
anthropic/claude-opus-4.8:high PASS 100% 271 in → 221 out (17 reasoning)
The bad commit is **e5c8297** ("reset config to zero"). **Reasoning:** - `d6a4132` (Initial commit): PASS - `16ef232` (set config value): PASS - `e5c8297` (reset config to zero): **FAIL** ← first failure - `70bcab5` (final config): FAIL (inherits the breakage) Since the tests pass up through `16ef232` and start failing at `e5c8297`, the commit that introduced the bug is **e5c8297 "reset config to zero"**. It likely reset a required config value to zero, breaking the build.
anthropic/claude-opus-4.8:high__json_schema PASS 100% 492 in → 15 out (0 reasoning)
a49b26b
JSON Schema Structured Output
(raw) {"commit":"a49b26b"}
anthropic/claude-opus-4.8:low PASS 100% 269 in → 72 out (0 reasoning)
ed1e40c The commit "reset config to zero" (ed1e40c) is the bad commit — the build passed at 428da96 and started failing at ed1e40c, continuing to fail in the subsequent commit.
anthropic/claude-opus-4.8:low__json_schema PASS 100% 494 in → 17 out (0 reasoning)
2dd3a4f
JSON Schema Structured Output
(raw) {"commit":"2dd3a4f"}
anthropic/claude-opus-4.8:max PASS 100% 277 in → 607 out (50 reasoning)
# Analysis Looking at the test results in chronological order (oldest → newest): | Commit | Subject | Result | |--------|---------|--------| | `09e0f1b` | Initial commit | ✅ PASS | | `87a9d56` | set config value | ✅ PASS | | `d062bb8` | reset config to zero | ❌ **FAIL** | | `a7d0dd0` | final config | ❌ FAIL | The build transitions from **PASS → FAIL** at commit `d062bb8`. Everything before it passes, and everything from it onward fails. This is the first bad commit that introduced the bug. ## Answer **`d062bb8`** — "reset config to zero" The commit subject strongly corroborates this: resetting the config value to zero likely introduced an invalid state (e.g., a divide-by-zero, a disabled required setting, or a zeroed-out parameter) that broke the build. The subsequent commit `a7d0dd0` inherits the same broken state, which is why it continues to fail.
anthropic/claude-opus-4.8:max__json_schema PASS 100% 488 in → 148 out (39 reasoning)
b0aedf3
JSON Schema Structured Output
(raw) {"commit": "b0aedf3"}
anthropic/claude-opus-4.8:medium PASS 100% 261 in → 164 out (17 reasoning)
The bad commit is **215a16b** ("reset config to zero"). This is the first commit where the build started failing: - `42898ca` → PASS - `c7a19fd` → PASS - `215a16b` → **FAIL** ← first failure (bad commit) - `9155473` → FAIL (inherits the breakage) Since the failure first appears at `215a16b` and all preceding commits pass, this is the commit that introduced the bug.
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 490 in → 15 out (0 reasoning)
dee2db2
JSON Schema Structured Output
(raw) {"commit": "dee2db2"}
anthropic/claude-opus-4.8:none PASS 100% 265 in → 62 out (0 reasoning)
The bad commit is **`2a69c87`** (`reset config to zero`). The tests pass through `ba4950f` and start failing at `2a69c87`, which is where the build broke.
anthropic/claude-opus-4.8:none__json_schema PASS 100% 488 in → 14 out (0 reasoning)
6d109f6
JSON Schema Structured Output
(raw) {"commit": "6d109f6"}
anthropic/claude-opus-4.8:xhigh PASS 100% 267 in → 373 out (56 reasoning)
## Analysis Looking at the test results in chronological order (oldest to newest): | Commit | Subject | Result | |--------|---------|--------| | `dc8f439` | Initial commit | ✅ PASS | | `9f5842a` | set config value | ✅ PASS | | `0c0d473` | reset config to zero | ❌ FAIL | | `a33514d` | final config | ❌ FAIL | ## Conclusion The build transitions from **PASS → FAIL** at commit `0c0d473`. Everything before it passes, and everything from that point onward fails (the failure persists into `a33514d`). **Bad commit: `0c0d473` — "reset config to zero"** This commit is the first one where the build breaks. The "reset config to zero" change likely set a configuration value to an invalid/zero state that the build or tests depend on being non-zero.
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 482 in → 13 out (0 reasoning)
d289924
JSON Schema Structured Output
(raw) {"commit": "d289924"}
anthropic/claude-sonnet-4.6:high PASS 100% 207 in → 63 out (0 reasoning)
The bad commit is **0e76a7f** This is the commit with the subject line "reset config to zero" — it's the first commit where the tests start failing (exit code 1), and all subsequent commits also fail, indicating this is where the bug was introduced.
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 381 in → 12 out (0 reasoning)
10c5ebc
JSON Schema Structured Output
(raw) {"commit":"10c5ebc"}
anthropic/claude-sonnet-4.6:low PASS 100% 215 in → 8 out (0 reasoning)
efac7a1
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 375 in → 11 out (0 reasoning)
b340c37
JSON Schema Structured Output
(raw) {"commit":"b340c37"}
anthropic/claude-sonnet-4.6:max PASS 100% 207 in → 343 out (99 reasoning)
## Analysis Looking at the test results ordered from oldest to newest: | Commit | Subject | Result | |--------|---------|--------| | `e5b631f` | Initial commit | ✅ PASS | | `91b6682` | set config value | ✅ PASS | | `448fc91` | reset config to zero | ❌ FAIL | | `d0870ee` | final config | ❌ FAIL | The build transitions from **PASS → FAIL** at commit `448fc91`. The subsequent commit `d0870ee` also fails because it inherits the bug introduced in the previous commit. ## Bad Commit **`448fc91`** — *"reset config to zero"* This commit reset a configuration value to zero, which likely caused a division-by-zero error, an invalid state, or a required non-zero value check to fail — breaking the build from that point forward.
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 383 in → 93 out (56 reasoning)
34f7dd2
JSON Schema Structured Output
(raw) {"commit":"34f7dd2"}
anthropic/claude-sonnet-4.6:medium PASS 100% 205 in → 13 out (0 reasoning)
The bad commit is **61704f2**.
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 373 in → 10 out (0 reasoning)
593174c
JSON Schema Structured Output
(raw) {"commit":"593174c"}
anthropic/claude-sonnet-4.6:none PASS 100% 211 in → 52 out (0 reasoning)
The bad commit is **54508ce** This is the commit ("reset config to zero") where the tests first started failing (exit code 1), and all subsequent commits continue to fail, indicating this is where the bug was introduced.
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 377 in → 10 out (0 reasoning)
87000de
JSON Schema Structured Output
(raw) {"commit":"87000de"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 209 in → 43 out (0 reasoning)
Based on the test results, the build was passing through commit `292eb96` and started failing at commit `28047b6`. The bad commit is: **28047b6**
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 373 in → 11 out (0 reasoning)
28f608a
JSON Schema Structured Output
(raw) {"commit":"28f608a"}
anthropic/claude-sonnet-5:high PASS 100% 267 in → 14 out (0 reasoning)
773f5e6 reset config to zero
anthropic/claude-sonnet-5:high__json_schema PASS 100% 488 in → 14 out (0 reasoning)
aea0533
JSON Schema Structured Output
(raw) {"commit":"aea0533"}
anthropic/claude-sonnet-5:low PASS 100% 265 in → 16 out (0 reasoning)
**30e90e8 reset config to zero**
anthropic/claude-sonnet-5:low__json_schema PASS 100% 490 in → 15 out (0 reasoning)
4fb642d
JSON Schema Structured Output
(raw) {"commit":"4fb642d"}
anthropic/claude-sonnet-5:medium PASS 100% 279 in → 9 out (0 reasoning)
4ed5a6f
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 496 in → 16 out (0 reasoning)
b38a9e3
JSON Schema Structured Output
(raw) {"commit":"b38a9e3"}
anthropic/claude-sonnet-5:none PASS 100% 261 in → 96 out (0 reasoning)
**a3a827c** (reset config to zero) This is the first commit where the test transitions from PASS to FAIL, indicating it introduced the breaking change. The subsequent commit `b31513b` continues to fail, consistent with the bug persisting after being introduced in `a3a827c`.
anthropic/claude-sonnet-5:none__json_schema PASS 100% 488 in → 14 out (0 reasoning)
2a9922a
JSON Schema Structured Output
(raw) {"commit":"2a9922a"}
anthropic/claude-sonnet-5:xhigh PASS 100% 273 in → 13 out (0 reasoning)
77694a8 reset config to zero
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 488 in → 15 out (0 reasoning)
1ff9566
JSON Schema Structured Output
(raw) {"commit":"1ff9566"}
arcee-ai/trinity-large-thinking:high PASS 100% 196 in → 462 out (430 reasoning)
The first commit that caused the test to fail is `d2d8545 reset config to zero`. This is the bad commit that introduced the bug.
arcee-ai/trinity-large-thinking:low PASS 100% 196 in → 407 out (402 reasoning)
8c94d81
arcee-ai/trinity-large-thinking:medium PASS 100% 198 in → 568 out (536 reasoning)
The bad commit is `8bd2aca` (subject line: "reset config to zero"), as it is the first commit where the tests fail.
arcee-ai/trinity-large-thinking:xhigh PASS 100% 188 in → 446 out (414 reasoning)
The first commit that causes a test failure is `79cc796` ("reset config to zero"). This indicates the bug was introduced in that commit.
arcee-ai/trinity-mini:high PASS 100% 190 in → 815 out (731 reasoning)
0b310c The bad commit is `0b310ce` (first 7 characters: `0b310c`), as it is the first commit where tests fail (`FAIL (exit code 1)`). All prior commits (including the initial commit and `set config value`) passed, indicating the bug was introduced in this commit. The subsequent `final config` commit also fails, but the root cause is the `reset config to zero` commit.
arcee-ai/trinity-mini:high__json_schema PASS 100% 194 in → 688 out (664 reasoning)
0d4f12d
JSON Schema Structured Output
(raw) { "commit": "0d4f12d" }
arcee-ai/trinity-mini:low PASS 100% 196 in → 1,386 out (1,363 reasoning)
9b4435e The test results show that the build passed for the first two commits (a56b75d and 0ec4edb), then failed for the third commit (9b4435e) and the fourth commit (2dacf78). Since the problem states there is only one bad commit that broke the build, the failure starting at commit 9b4435e indicates this is the problematic commit. The subsequent failure at 2dacf78 is likely a result of the initial bug not being fixed, but the root cause is the commit introducing the issue.
arcee-ai/trinity-mini:low__json_schema PASS 100% 196 in → 471 out (473 reasoning)
a3f940d
JSON Schema Structured Output
(raw) { "commit": "a3f940d" }
arcee-ai/trinity-mini:medium PASS 100% 200 in → 663 out (709 reasoning)
4ff7352 (reset config to zero)
arcee-ai/trinity-mini:medium__json_schema PASS 100% 192 in → 394 out (414 reasoning)
4a9b033
JSON Schema Structured Output
(raw) { "commit": "4a9b033" }
arcee-ai/trinity-mini:xhigh PASS 100% 194 in → 500 out (438 reasoning)
1940cee The bad commit is `1940cee` (reset config to zero), as it is the first commit where the test fails (exit code 1). The subsequent commit (`23c2eb8`) also fails, but the bug was introduced in `1940cee`.
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 198 in → 672 out (696 reasoning)
97b09b0
JSON Schema Structured Output
(raw) { "commit": "97b09b0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 189 in → 167 out (153 reasoning)
b7c169a
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 185 in → 243 out (213 reasoning)
0675409
JSON Schema Structured Output
(raw) { "commit": "0675409" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 264 in → 156 out (152 reasoning)
20719aa
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 189 in → 11 out (0 reasoning)
5370b4f
JSON Schema Structured Output
(raw) {"commit": "5370b4f"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 279 in → 267 out (262 reasoning)
9407d37
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 285 in → 208 out (188 reasoning)
4b5a06a
JSON Schema Structured Output
(raw) { "commit": "4b5a06a" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 189 in → 230 out (224 reasoning)
9716d18
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 187 in → 294 out (254 reasoning)
817af10
JSON Schema Structured Output
(raw) { "commit": "817af10" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 187 in → 35 out (0 reasoning)
The bad commit is **3bd9118** (reset config to zero), since this is the first commit where the test fails, and the failure persists in subsequent commits.
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 183 in → 15 out (0 reasoning)
a933efb
JSON Schema Structured Output
(raw) { "commit": "a933efb" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 266 in → 159 out (153 reasoning)
a8dcc50
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 189 in → 203 out (170 reasoning)
76d4505
JSON Schema Structured Output
(raw) {"commit":"76d4505"}
deepseek/deepseek-v4-flash:high PASS 100% 187 in → 145 out (139 reasoning)
97d337f
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 195 in → 138 out (123 reasoning)
6f27357
JSON Schema Structured Output
(raw) { "commit": "6f27357" }
deepseek/deepseek-v4-flash:low PASS 100% 183 in → 102 out (90 reasoning)
15071da
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 195 in → 105 out (82 reasoning)
4a70e2d
JSON Schema Structured Output
(raw) { "commit": "4a70e2d" }
deepseek/deepseek-v4-flash:medium PASS 100% 195 in → 88 out (81 reasoning)
138fe8f
deepseek/deepseek-v4-flash:none PASS 100% 193 in → 73 out (0 reasoning)
Based on the test results, the build passes for commits `5f311eb` (Initial commit) and `7cc011e` (set config value), but fails starting from `7497b1b` (reset config to zero). This means the bug was introduced in commit `7497b1b`. **7497b1b**
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 195 in → 12 out (0 reasoning)
9c6770c
JSON Schema Structured Output
(raw) { "commit": "9c6770c" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 278 in → 206 out (200 reasoning)
dff66bc
deepseek/deepseek-v4-pro:high PASS 100% 187 in → 149 out (145 reasoning)
838457b
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 191 in → 255 out (242 reasoning)
fc59873
JSON Schema Structured Output
(raw) { "commit": "fc59873" }
deepseek/deepseek-v4-pro:low PASS 100% 193 in → 273 out (266 reasoning)
34a8b13
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 185 in → 170 out (157 reasoning)
61fed01
JSON Schema Structured Output
(raw) { "commit": "61fed01" }
deepseek/deepseek-v4-pro:medium PASS 100% 191 in → 279 out (270 reasoning)
8e6d1c2
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 382 in → 101 out (89 reasoning)
55abf2c
JSON Schema Structured Output
(raw) {"commit": "55abf2c"}
deepseek/deepseek-v4-pro:none PASS 100% 189 in → 56 out (0 reasoning)
Based on the test results, the build was passing until commit `891127d reset config to zero`, which introduced the first failure. Since it’s the earliest commit where the test starts failing, that is the bad commit. **Bad commit:** `891127d`
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 193 in → 10 out (0 reasoning)
835cc3b
JSON Schema Structured Output
(raw) {"commit": "835cc3b"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 268 in → 365 out (309 reasoning)
The bug was introduced in commit `355c99c` with the subject "reset config to zero", as it is the first commit where the test fails. The previous commit (`cc4674d`) passed. **Bad commit hash:** `355c99c`
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 266 in → 252 out (241 reasoning)
65f6601
JSON Schema Structured Output
(raw) {"commit": "65f6601"}
google/gemini-3-flash-preview:high PASS 100% 215 in → 305 out (298 reasoning)
19990b2
google/gemini-3-flash-preview:high__json_schema PASS 100% 272 in → 682 out (667 reasoning)
7f9beb6
JSON Schema Structured Output
(raw) { "commit": "7f9beb6" }
google/gemini-3-flash-preview:low PASS 100% 213 in → 249 out (242 reasoning)
82c9e51
google/gemini-3-flash-preview:low__json_schema PASS 100% 270 in → 197 out (186 reasoning)
63b4682
JSON Schema Structured Output
(raw) {"commit":"63b4682"}
google/gemini-3-flash-preview:medium PASS 100% 215 in → 340 out (333 reasoning)
58f40c1
google/gemini-3-flash-preview:medium__json_schema PASS 100% 266 in → 199 out (183 reasoning)
7ab2271
JSON Schema Structured Output
(raw) { "commit": "7ab2271" }
google/gemini-3-flash-preview:xhigh PASS 100% 213 in → 273 out (267 reasoning)
758bcf5
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 214 in → 120 out (109 reasoning)
8d9c273
JSON Schema Structured Output
(raw) {"commit":"8d9c273"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 211 in → 493 out (417 reasoning)
The bad commit is **c9c339c** (reset config to zero). **Reasoning:** The commit history shows that the test passed at `d2a3cd2` but failed at `c9c339c`. Therefore, the changes introduced in `c9c339c` are what caused the build failure.
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 208 in → 341 out (332 reasoning)
eeae931
JSON Schema Structured Output
(raw) {"commit":"eeae931"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 214 in → 199 out (112 reasoning)
The bad commit is **a42981e** (subject: **reset config to zero**). **Reasoning:** The test results show that the build passed on `2a64f4c` but failed on `a42981e`. Since `a42981e` is the first commit where the failure appeared, it is the one that introduced the bug.
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 272 in → 117 out (101 reasoning)
b01877e
JSON Schema Structured Output
(raw) { "commit": "b01877e" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 214 in → 247 out (241 reasoning)
dd7e9a8
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 216 in → 228 out (216 reasoning)
177f4a5
JSON Schema Structured Output
(raw) {"commit": "177f4a5"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 209 in → 567 out (530 reasoning)
The bad commit is **efed42d** (subject: **reset config to zero**). This is the first commit where the test status changed from PASS to FAIL.
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 270 in → 285 out (273 reasoning)
58d96c7
JSON Schema Structured Output
(raw) {"commit": "58d96c7"}
google/gemini-3.1-pro-preview:high PASS 100% 211 in → 149 out (130 reasoning)
The bad commit is **f6545f1** ("reset config to zero").
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 274 in → 166 out (156 reasoning)
cf4a13b
JSON Schema Structured Output
(raw) {"commit":"cf4a13b"}
google/gemini-3.1-pro-preview:low PASS 100% 211 in → 125 out (104 reasoning)
The bad commit that introduced the bug is **8baa93d** ("reset config to zero").
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 272 in → 49 out (34 reasoning)
4cf9ee7
JSON Schema Structured Output
(raw) { "commit": "4cf9ee7" }
google/gemini-3.1-pro-preview:medium PASS 100% 209 in → 210 out (159 reasoning)
The bad commit that introduced the bug is **7fb5111** (or "reset config to zero"). Looking at the test results, the build was passing until this commit was introduced, shifting the test result from PASS to FAIL.
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 270 in → 124 out (112 reasoning)
3955085
JSON Schema Structured Output
(raw) {"commit": "3955085"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 215 in → 154 out (148 reasoning)
07aeb58
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 268 in → 142 out (131 reasoning)
98bb0f9
JSON Schema Structured Output
(raw) {"commit": "98bb0f9"}
google/gemini-3.5-flash-lite:high PASS 100% 214 in → 400 out (393 reasoning)
a697191
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 274 in → 499 out (487 reasoning)
349a50a
JSON Schema Structured Output
(raw) {"commit": "349a50a"}
google/gemini-3.5-flash-lite:low PASS 100% 211 in → 125 out (0 reasoning)
To identify the bad commit, we look at the test results in chronological order (oldest to newest): 1. `6d212f2 Initial commit`: PASS 2. `7547133 set config value`: PASS 3. `ced30a6 reset config to zero`: FAIL Since the build passed in commit `7547133` but failed in the very next commit `ced30a6`, the bug was introduced in `ced30a6`. **ced30a6** (or "reset config to zero")
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 208 in → 15 out (0 reasoning)
c4cdfd3
JSON Schema Structured Output
(raw) { "commit": "c4cdfd3" }
google/gemini-3.5-flash-lite:max PASS 100% 211 in → 352 out (346 reasoning)
294b8ac
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 212 in → 485 out (475 reasoning)
db3a8f2
JSON Schema Structured Output
(raw) {"commit":"db3a8f2"}
google/gemini-3.5-flash-lite:medium PASS 100% 213 in → 191 out (186 reasoning)
90efaa5
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 210 in → 287 out (277 reasoning)
beccb19
JSON Schema Structured Output
(raw) {"commit": "beccb19"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 211 in → 354 out (347 reasoning)
9a626f9
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 216 in → 479 out (468 reasoning)
04a5859
JSON Schema Structured Output
(raw) {"commit":"04a5859"}
google/gemini-3.5-flash:high PASS 100% 211 in → 322 out (316 reasoning)
0a091bd
google/gemini-3.5-flash:high__json_schema PASS 100% 276 in → 370 out (358 reasoning)
3b5f82e
JSON Schema Structured Output
(raw) {"commit": "3b5f82e"}
google/gemini-3.5-flash:low PASS 100% 213 in → 80 out (73 reasoning)
19b52e7
google/gemini-3.5-flash:low__json_schema PASS 100% 272 in → 114 out (98 reasoning)
74dcb0f
JSON Schema Structured Output
(raw) { "commit": "74dcb0f" }
google/gemini-3.5-flash:medium PASS 100% 215 in → 194 out (187 reasoning)
8a7762e
google/gemini-3.5-flash:medium__json_schema PASS 100% 274 in → 209 out (199 reasoning)
0eb9d97
JSON Schema Structured Output
(raw) {"commit":"0eb9d97"}
google/gemini-3.5-flash:xhigh PASS 100% 211 in → 236 out (230 reasoning)
bbfd9c0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 270 in → 347 out (337 reasoning)
b8afc8f
JSON Schema Structured Output
(raw) {"commit":"b8afc8f"}
google/gemini-3.6-flash:high PASS 100% 216 in → 284 out (277 reasoning)
800a636
google/gemini-3.6-flash:high__json_schema PASS 100% 274 in → 397 out (386 reasoning)
064f58b
JSON Schema Structured Output
(raw) {"commit":"064f58b"}
google/gemini-3.6-flash:low PASS 100% 212 in → 136 out (114 reasoning)
The commit that introduced the bug is **c30c03e** (reset config to zero).
google/gemini-3.6-flash:low__json_schema PASS 100% 214 in → 101 out (84 reasoning)
d4684b1
JSON Schema Structured Output
(raw) { "commit": "d4684b1" }
google/gemini-3.6-flash:max PASS 100% 209 in → 311 out (298 reasoning)
**fb3ab0a** (reset config to zero)
google/gemini-3.6-flash:max__json_schema PASS 100% 212 in → 354 out (344 reasoning)
3140ab5
JSON Schema Structured Output
(raw) {"commit":"3140ab5"}
google/gemini-3.6-flash:medium PASS 100% 216 in → 258 out (225 reasoning)
The bad commit that introduced the bug is **`59a3d9d`** (or subject line: **`reset config to zero`**).
google/gemini-3.6-flash:medium__json_schema PASS 100% 212 in → 256 out (245 reasoning)
a8247f8
JSON Schema Structured Output
(raw) {"commit":"a8247f8"}
google/gemini-3.6-flash:xhigh PASS 100% 212 in → 245 out (238 reasoning)
49d0071
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 274 in → 319 out (308 reasoning)
9098f10
JSON Schema Structured Output
(raw) {"commit":"9098f10"}
google/gemma-4-26b-a4b-it:high PASS 100% 227 in → 328 out (320 reasoning)
3559c11
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 227 in → 373 out (242 reasoning)
fdc7122
JSON Schema Structured Output
(raw) { "commit": "fdc7122" }
google/gemma-4-26b-a4b-it:low PASS 100% 222 in → 165 out (103 reasoning)
fd5ce66
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 227 in → 435 out (302 reasoning)
1b87d68
JSON Schema Structured Output
(raw) {"commit": "1b87d68"}
google/gemma-4-26b-a4b-it:medium PASS 100% 231 in → 365 out (266 reasoning)
96c31bc
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 231 in → 356 out (227 reasoning)
a964387
JSON Schema Structured Output
(raw) { "commit": "a964387" }
google/gemma-4-26b-a4b-it:none PASS 100% 222 in → 7 out (0 reasoning)
e22ad7e
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 224 in → 16 out (0 reasoning)
bc8c08e
JSON Schema Structured Output
(raw) { "commit": "bc8c08e" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 229 in → 357 out (259 reasoning)
663d8ca
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 233 in → 257 out (156 reasoning)
5185293
JSON Schema Structured Output
(raw) { "commit": "5185293" }
google/gemma-4-31b-it:high PASS 100% 231 in → 208 out (140 reasoning)
c23498e
google/gemma-4-31b-it:high__json_schema PASS 100% 229 in → 213 out (141 reasoning)
fbff938
JSON Schema Structured Output
(raw) {"commit": "fbff938"}
google/gemma-4-31b-it:low PASS 100% 231 in → 178 out (121 reasoning)
8b98e11
google/gemma-4-31b-it:low__json_schema PASS 100% 222 in → 167 out (102 reasoning)
8fd3e18
JSON Schema Structured Output
(raw) {"commit": "8fd3e18"}
google/gemma-4-31b-it:medium PASS 100% 227 in → 140 out (93 reasoning)
c1a8c8f
google/gemma-4-31b-it:medium__json_schema PASS 100% 222 in → 170 out (126 reasoning)
aa6cff8
JSON Schema Structured Output
(raw) {"commit": "aa6cff8"}
google/gemma-4-31b-it:none PASS 100% 226 in → 8 out (0 reasoning)
725e521
google/gemma-4-31b-it:none__json_schema PASS 100% 224 in → 12 out (0 reasoning)
851c6da
JSON Schema Structured Output
(raw) {"commit":"851c6da"}
google/gemma-4-31b-it:xhigh PASS 100% 222 in → 205 out (138 reasoning)
1bc4cf7
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 224 in → 206 out (137 reasoning)
efcccd8
JSON Schema Structured Output
(raw) {"commit": "efcccd8"}
ibm-granite/granite-4.1-8b PASS 100% 208 in → 41 out
The bad commit that introduced the bug is the commit with hash `f11be15`. It is the first commit that caused a test failure. Therefore, the answer is `f11be15`.
117e596
JSON Schema Structured Output
(raw) {"commit":"117e596"}
inclusionai/ling-2.6-flash PASS 100% 235 in → 8 out
4782293
inclusionai/ling-2.6-flash__json_schema PASS 100% 233 in → 17 out
1fbc789
JSON Schema Structured Output
(raw) { "commit": "1fbc789" }
liquid/lfm-2-24b-a2b PASS 100% 206 in → 186 out
Let's analyze step-by-step: - **Commits (newest first):** 1. `f62a028` – Final config (FAIL) 2. `ac493d2` – Reset config to zero (FAIL) 3. `3403f48` – Set config value (PASS) 4. `8fed8b3` – Initial commit (PASS) - **Test results:** - All commits before `f62a028` passed. - Both `ac493d2` and `f62a028` failed. Since the build fails only after `f62a028`, and both previous commits passed, it means the **bad commit** is the most recent one: **Commit hash:** `f62a028` **Answer:** `f62a028`
minimax/minimax-m2.5:high PASS 100% 228 in → 233 out (228 reasoning)
118ce6b
minimax/minimax-m2.5:high__json_schema PASS 100% 221 in → 286 out (273 reasoning)
16aff1c
JSON Schema Structured Output
(raw) { "commit": "16aff1c" }
minimax/minimax-m2.5:low PASS 100% 225 in → 263 out (248 reasoning)
d2e1030
minimax/minimax-m2.5:low__json_schema PASS 100% 227 in → 311 out (301 reasoning)
c8da2cb
JSON Schema Structured Output
(raw) { "commit": "c8da2cb" }
minimax/minimax-m2.5:medium PASS 100% 221 in → 174 out (166 reasoning)
3675a90
minimax/minimax-m2.5:medium__json_schema PASS 100% 225 in → 274 out (248 reasoning)
4bc2ffa
JSON Schema Structured Output
(raw) { "commit": "4bc2ffa" }
minimax/minimax-m2.5:xhigh PASS 100% 225 in → 208 out (182 reasoning)
9e04a7b
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 223 in → 317 out (301 reasoning)
6e206b7
JSON Schema Structured Output
(raw) { "commit": "6e206b7" }
minimax/minimax-m2.7:high PASS 100% 226 in → 256 out (244 reasoning)
62ae6c3 (reset config to zero)
minimax/minimax-m2.7:high__json_schema PASS 100% 224 in → 267 out (266 reasoning)
4258662
JSON Schema Structured Output
(raw) { "commit": "4258662" }
minimax/minimax-m2.7:low PASS 100% 222 in → 357 out (345 reasoning)
**769cee6** (reset config to zero)
minimax/minimax-m2.7:low__json_schema PASS 100% 368 in → 441 out (425 reasoning)
1af5a6a
JSON Schema Structured Output
(raw) { "commit": "1af5a6a" }
minimax/minimax-m2.7:medium PASS 100% 234 in → 274 out (262 reasoning)
6b1d7e0
minimax/minimax-m2.7:medium__json_schema PASS 100% 360 in → 190 out (180 reasoning)
76cc0ff
JSON Schema Structured Output
(raw) {"commit":"76cc0ff"}
minimax/minimax-m2.7:xhigh PASS 100% 227 in → 318 out (311 reasoning)
c81ad5d
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 360 in → 641 out (631 reasoning)
51f930a
JSON Schema Structured Output
(raw) {"commit":"51f930a"}
minimax/minimax-m3:high PASS 100% 363 in → 93 out (80 reasoning)
57cc098
minimax/minimax-m3:high__json_schema PASS 100% 348 in → 12 out (0 reasoning)
1ffadd1
JSON Schema Structured Output
(raw) { "commit": "1ffadd1"}
minimax/minimax-m3:low PASS 100% 369 in → 139 out (123 reasoning)
233e33e
minimax/minimax-m3:low__json_schema PASS 100% 344 in → 9 out (0 reasoning)
9795569
JSON Schema Structured Output
(raw) {"commit": "9795569"}
minimax/minimax-m3:medium PASS 100% 359 in → 50 out (54 reasoning)
7021bcf
minimax/minimax-m3:medium__json_schema PASS 100% 348 in → 10 out (0 reasoning)
d100f75
JSON Schema Structured Output
(raw) {"commit": "d100f75"}
minimax/minimax-m3:xhigh PASS 100% 363 in → 158 out (89 reasoning)
**807c8f3** The commit "reset config to zero" (807c8f3) is the first commit that failed testing, making it the commit that introduced the bug. The previous commit "set config value" (b76ffac) passed, and all subsequent commits continued to fail.
minimax/minimax-m3:xhigh__json_schema PASS 100% 350 in → 9 out (0 reasoning)
0198633
JSON Schema Structured Output
(raw) {"commit": "0198633"}
mistralai/devstral-2512 PASS 100% 213 in → 96 out
Based on the test results, the build started failing at commit `bc9dd84 reset config to zero` and continued to fail in the subsequent commit `791e706 final config`. Since the build was passing at `a2d75d0 set config value` and started failing at `bc9dd84 reset config to zero`, the bad commit is: `bc9dd84` (or "reset config to zero")
mistralai/devstral-2512__json_schema PASS 100% 205 in → 12 out
f9cf059
JSON Schema Structured Output
(raw) {"commit": "f9cf059"}
mistralai/mistral-medium-3-5:high PASS 100% 225 in → 394 out (273 reasoning)
The build passed for commit `ddb6e97` but failed for the next commit. The first failing commit is `dab4139`, which introduced the regression. dab4139
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 211 in → 359 out (289 reasoning)
e86584a
JSON Schema Structured Output
(raw) {"commit": "e86584a"}
mistralai/mistral-medium-3-5:low PASS 100% 225 in → 637 out (434 reasoning)
The bad commit is the first one in the sequence where tests transition from PASS to FAIL. - `b698d81` (Initial commit): PASS - `c85721b` (set config value): PASS - `ca689f3` (reset config to zero): **FAIL** (first failure) - `ec89379` (final config): FAIL (consequent failure) Since the failure starts at `ca689f3`, this commit introduced the bug. ca689f3
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 209 in → 378 out (324 reasoning)
fdbac11
JSON Schema Structured Output
(raw) {"commit": "fdbac11"}
mistralai/mistral-medium-3-5:medium PASS 100% 229 in → 484 out (366 reasoning)
The bad commit is the first one that caused the test to fail. Based on the test results, the commit `1682972` (subject: "reset config to zero") introduced the bug. 1682972
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 215 in → 380 out (293 reasoning)
935e436
JSON Schema Structured Output
(raw) {"commit": "935e436"}
mistralai/mistral-medium-3-5:none PASS 100% 225 in → 100 out (0 reasoning)
The git bisect approach would identify the first failing commit as the one that introduced the bug. In this case: - The commit `16ce085 reset config to zero` is the first one that fails (exit code 1) - All previous commits (`b3085b5` and `d631d81`) passed Therefore, the bad commit is **`16ce085`** (or "reset config to zero").
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 211 in → 12 out (0 reasoning)
ed44c77
JSON Schema Structured Output
(raw) {"commit": "ed44c77"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 225 in → 520 out (398 reasoning)
8181a1c
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 213 in → 373 out (294 reasoning)
d16c947
JSON Schema Structured Output
(raw) {"commit": "d16c947"}
moonshotai/kimi-k2.6:high PASS 100% 193 in → 353 out (348 reasoning)
**99e9055**
moonshotai/kimi-k2.6:high__json_schema PASS 100% 190 in → 655 out (273 reasoning)
da69562
JSON Schema Structured Output
(raw) { "commit": "da69562" }
moonshotai/kimi-k2.6:low PASS 100% 193 in → 491 out (433 reasoning)
6b30d00
moonshotai/kimi-k2.6:low__json_schema PASS 100% 194 in → 366 out (331 reasoning)
1facc0e
JSON Schema Structured Output
(raw) { "commit": "1facc0e" }
moonshotai/kimi-k2.6:medium PASS 100% 201 in → 182 out (150 reasoning)
4a8ff1f
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 186 in → 486 out (287 reasoning)
cf07e22
JSON Schema Structured Output
(raw) { "commit": "cf07e22" }
moonshotai/kimi-k2.6:xhigh PASS 100% 189 in → 429 out (435 reasoning)
1d9dba1
moonshotai/kimi-k2.7-code:high PASS 100% 195 in → 169 out (156 reasoning)
**bbf35c5** — *reset config to zero*
moonshotai/kimi-k2.7-code:low PASS 100% 185 in → 112 out (95 reasoning)
2607996
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 284 in → 184 out (172 reasoning)
645b3e9
JSON Schema Structured Output
(raw) {"commit": "645b3e9"}
moonshotai/kimi-k2.7-code:medium PASS 100% 193 in → 178 out (171 reasoning)
c97ab5d
moonshotai/kimi-k2.7-code:xhigh PASS 100% 197 in → 76 out (62 reasoning)
4c2d2dc (reset config to zero)
moonshotai/kimi-k3:high PASS 100% 270 in → 106 out (86 reasoning)
3b0bc35
moonshotai/kimi-k3:high__json_schema PASS 100% 272 in → 228 out (55 reasoning)
**Bad commit: `7889df2` — "reset config to zero"** **Reasoning:** - `0e3cdbb` (Initial commit): PASS - `c6b46e6` (set config value): PASS — still good - `7889df2` (reset config to zero): **FAIL** — first failing commit - `c531097` (final config): FAIL — still broken Since `c6b46e6` passes and its immediate child `7889df2` is the first commit to fail, `7889df2` ("reset config to zero") is the commit that introduced the bug. The subsequent commit `c531097` fails because it inherits the breakage — it didn't fix it.
moonshotai/kimi-k3:low PASS 100% 266 in → 293 out (130 reasoning)
**The bad commit is `c80eb9c` ("reset config to zero")** **Reasoning:** - `cad9556` (Initial commit): PASS ✅ - `fdf99ed` (set config value): PASS ✅ - `c80eb9c` (reset config to zero): FAIL ❌ ← **first failing commit** - `75f5230` (final config): FAIL ❌ Since the build passed at `fdf99ed` and began failing at `c80eb9c`, this is the commit that introduced the bug. The later commit `75f5230` simply inherited the broken state — a classic bisect result pointing to **`c80eb9c`**.
moonshotai/kimi-k3:low__json_schema PASS 100% 378 in → 78 out (54 reasoning)
90b0de4
JSON Schema Structured Output
(raw) {"commit":"90b0de4"}
moonshotai/kimi-k3:max PASS 100% 264 in → 182 out (78 reasoning)
**Bad commit: `d524dc6` ("reset config to zero")** Reasoning: The build passed on commits `946b227` and `0172a06`, but first fails at `d524dc6` and remains failing at `246eadf`. Since a bisect identifies the bad commit as the first commit where the test transitions from PASS to FAIL, the bug was introduced in **`d524dc6`**.
moonshotai/kimi-k3:max__json_schema PASS 100% 274 in → 284 out (87 reasoning)
**Bad commit: `c7ad9d9` ("reset config to zero")** **Reasoning:** Walking the history from oldest to newest: - `38136e9` (Initial commit) — PASS ✅ - `3ae9f49` (set config value) — PASS ✅ - `c7ad9d9` (reset config to zero) — FAIL ❌ ← first failure - `92d4908` (final config) — FAIL ❌ (inherits the broken state) Since `c7ad9d9` is the first commit where the build transitions from passing to failing, it introduced the bug. The later commit `92d4908` still fails, but that's only because it builds on top of the already-broken state — a `git bisect` would converge on `c7ad9d9` as the culprit.
moonshotai/kimi-k3:medium PASS 100% 274 in → 377 out (179 reasoning)
**Bad commit: `a57486b` (reset config to zero)** **Reasoning:** Looking at the test results chronologically: - `de4b64e` (Initial commit) → PASS - `d5d6616` (set config value) → PASS - `a57486b` (reset config to zero) → **FAIL** ← first failure - `f7c6424` (final config) → FAIL The build transitioned from passing to failing at commit `a57486b` ("reset config to zero"). Since `d5d6616` passes and every commit from `a57486b` onward fails, `a57486b` is the commit that introduced the bug. Resetting the config value to zero likely broke something the build depends on (e.g., division by the config value, or a required non-zero setting).
moonshotai/kimi-k3:medium__json_schema PASS 100% 372 in → 112 out (88 reasoning)
ae51b2a
JSON Schema Structured Output
(raw) {"commit":"ae51b2a"}
moonshotai/kimi-k3:xhigh PASS 100% 276 in → 358 out (131 reasoning)
**Bad commit: `df1a4f9`** — "reset config to zero" **Reasoning:** Using a bisect-style analysis of the test results: | Commit | Result | |--------|--------| | 8a30ef0 (Initial commit) | PASS ✅ | | e5345a9 (set config value) | PASS ✅ | | **df1a4f9 (reset config to zero)** | **FAIL ❌** | | fc0367c (final config) | FAIL ❌ | The last known-good commit is `e5345a9`, and the first known-bad commit is `df1a4f9`. Since the build passes at `e5345a9` and starts failing at `df1a4f9` (and remains broken in its descendant `fc0367c`), the bug was introduced by commit **`df1a4f9`** ("reset config to zero") — likely the zeroed-out config value violates an assumption the build/tests depend on.
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 264 in → 206 out (63 reasoning)
**Bad commit: `c242695`** — "reset config to zero" **Reasoning:** - `6c3ef45` (Initial commit): PASS - `832d270` (set config value): PASS - `c242695` (reset config to zero): **FAIL** ← first failure - `edfdb39` (final config): FAIL (still broken, but the bug was already introduced) Since the build passes at `832d270` and first fails at `c242695`, the commit `c242695` ("reset config to zero") is the one that introduced the bug.
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 226 in → 307 out (250 reasoning)
74fe0c9
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 222 in → 393 out (357 reasoning)
db3786f
JSON Schema Structured Output
(raw) { "commit": "db3786f" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 226 in → 387 out (281 reasoning)
The first commit that caused thebuild to fail is **922a532**. This commit introduced the failure (exit code 1) in the "reset config to zero" test. Subsequent commits (e.g., 041f0ee) inherit the broken state and therefore also fail, but the root cause originates from **922a532**. **Answer:** `922a532` (or the commit subject “reset config to zero”).
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 218 in → 635 out (666 reasoning)
ede4fab
JSON Schema Structured Output
(raw) { "commit": "ede4fab" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 226 in → 254 out (203 reasoning)
f77a190
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 222 in → 269 out (220 reasoning)
9dc2018
JSON Schema Structured Output
(raw) { "commit": "9dc2018" }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 226 in → 13 out (0 reasoning)
02d72eb
JSON Schema Structured Output
(raw) { "commit":"02d72eb" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 226 in → 317 out (275 reasoning)
060e4a1
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 226 in → 3,173 out (2,744 reasoning)
9553a5b
JSON Schema Structured Output
(raw) { "commit": "9553a5b" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 220 in → 182 out (155 reasoning)
3e25021
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 226 in → 135 out (102 reasoning)
3e598f8
JSON Schema Structured Output
(raw) { "commit": "3e598f8" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 224 in → 186 out (156 reasoning)
d7178f4
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 226 in → 173 out (140 reasoning)
9a287a9
JSON Schema Structured Output
(raw) { "commit": "9a287a9" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 226 in → 144 out (130 reasoning)
fe49a0b
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 226 in → 112 out (96 reasoning)
49d0f3b
JSON Schema Structured Output
(raw) {"commit": "49d0f3b"}
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 222 in → 11 out (0 reasoning)
74757ee reset config to zero
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 224 in → 16 out (0 reasoning)
d560581
JSON Schema Structured Output
(raw) { "commit": "d560581" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 222 in → 151 out (121 reasoning)
c6cc5f3
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 222 in → 338 out (269 reasoning)
e247023
JSON Schema Structured Output
(raw) { "commit": "e247023" }
openai/gpt-5.4-mini:high PASS 100% 189 in → 105 out (94 reasoning)
56bc3d4
openai/gpt-5.4-mini:high__json_schema PASS 100% 232 in → 86 out (67 reasoning)
5d87599
JSON Schema Structured Output
(raw) {"commit":"5d87599"}
openai/gpt-5.4-mini:low PASS 100% 195 in → 57 out (45 reasoning)
0aa2e9e
openai/gpt-5.4-mini:low__json_schema PASS 100% 240 in → 60 out (39 reasoning)
4cbd0d7
JSON Schema Structured Output
(raw) {"commit":"4cbd0d7"}
openai/gpt-5.4-mini:medium PASS 100% 191 in → 51 out (40 reasoning)
b7e3735
openai/gpt-5.4-mini:medium__json_schema PASS 100% 234 in → 82 out (61 reasoning)
6b5aa9a
JSON Schema Structured Output
(raw) {"commit":"6b5aa9a"}
openai/gpt-5.4-mini:none PASS 100% 189 in → 8 out (0 reasoning)
5825e45
openai/gpt-5.4-mini:none__json_schema PASS 100% 228 in → 17 out (0 reasoning)
33dbd40
JSON Schema Structured Output
(raw) {"commit":"33dbd40"}
openai/gpt-5.4-mini:xhigh PASS 100% 193 in → 109 out (99 reasoning)
464c04b
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 228 in → 150 out (131 reasoning)
deeaa0d
JSON Schema Structured Output
(raw) {"commit":"deeaa0d"}
openai/gpt-5.4-nano:high PASS 100% 193 in → 44 out (33 reasoning)
0b4909f
openai/gpt-5.4-nano:high__json_schema PASS 100% 236 in → 74 out (54 reasoning)
237a2f0
JSON Schema Structured Output
(raw) {"commit":"237a2f0"}
openai/gpt-5.4-nano:low PASS 100% 191 in → 28 out (18 reasoning)
cc6a308
openai/gpt-5.4-nano:low__json_schema PASS 100% 240 in → 42 out (22 reasoning)
24d81b7
JSON Schema Structured Output
(raw) {"commit":"24d81b7"}
openai/gpt-5.4-nano:medium PASS 100% 191 in → 33 out (23 reasoning)
a720e67
openai/gpt-5.4-nano:medium__json_schema PASS 100% 236 in → 41 out (21 reasoning)
0d7e868
JSON Schema Structured Output
(raw) {"commit":"0d7e868"}
openai/gpt-5.4-nano:none PASS 100% 193 in → 12 out (0 reasoning)
b37230a reset config to zero
openai/gpt-5.4-nano:none__json_schema PASS 100% 240 in → 18 out (0 reasoning)
b6e6007
JSON Schema Structured Output
(raw) {"commit":"b6e6007"}
openai/gpt-5.4-nano:xhigh PASS 100% 189 in → 102 out (92 reasoning)
89c316e
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 228 in → 117 out (98 reasoning)
682d01e
JSON Schema Structured Output
(raw) {"commit":"682d01e"}
openai/gpt-5.4:high PASS 100% 189 in → 63 out (54 reasoning)
46ec106
openai/gpt-5.4:high__json_schema PASS 100% 236 in → 53 out (32 reasoning)
c5cc2b7
JSON Schema Structured Output
(raw) {"commit":"c5cc2b7"}
openai/gpt-5.4:low PASS 100% 187 in → 23 out (14 reasoning)
adce435
openai/gpt-5.4:low__json_schema PASS 100% 236 in → 43 out (22 reasoning)
4a4a2fc
JSON Schema Structured Output
(raw) {"commit":"4a4a2fc"}
openai/gpt-5.4:medium PASS 100% 191 in → 54 out (42 reasoning)
1c26c8d
openai/gpt-5.4:medium__json_schema PASS 100% 234 in → 48 out (27 reasoning)
d3b60e5
JSON Schema Structured Output
(raw) {"commit":"d3b60e5"}
openai/gpt-5.4:none PASS 100% 187 in → 8 out (0 reasoning)
7b20841
openai/gpt-5.4:none__json_schema PASS 100% 230 in → 17 out (0 reasoning)
2cacb62
JSON Schema Structured Output
(raw) {"commit":"2cacb62"}
openai/gpt-5.4:xhigh PASS 100% 191 in → 85 out (75 reasoning)
3182aa6
openai/gpt-5.4:xhigh__json_schema PASS 100% 226 in → 139 out (121 reasoning)
e518593
JSON Schema Structured Output
(raw) {"commit":"e518593"}
openai/gpt-5.5:high PASS 100% 189 in → 42 out (32 reasoning)
d9eb770
openai/gpt-5.5:high__json_schema PASS 100% 236 in → 76 out (57 reasoning)
238f2ad
JSON Schema Structured Output
(raw) {"commit":"238f2ad"}
openai/gpt-5.5:low PASS 100% 195 in → 28 out (18 reasoning)
9de6883
openai/gpt-5.5:low__json_schema PASS 100% 230 in → 18 out (0 reasoning)
caf2f4b
JSON Schema Structured Output
(raw) {"commit":"caf2f4b"}
openai/gpt-5.5:medium PASS 100% 191 in → 28 out (18 reasoning)
e822ce2
openai/gpt-5.5:medium__json_schema PASS 100% 232 in → 18 out (0 reasoning)
77e21e6
JSON Schema Structured Output
(raw) {"commit":"77e21e6"}
openai/gpt-5.5:none PASS 100% 189 in → 8 out (0 reasoning)
27705e0
openai/gpt-5.5:none__json_schema PASS 100% 234 in → 17 out (0 reasoning)
e33a458
JSON Schema Structured Output
(raw) {"commit":"e33a458"}
openai/gpt-5.5:xhigh PASS 100% 189 in → 54 out (45 reasoning)
9837594
openai/gpt-5.5:xhigh__json_schema PASS 100% 230 in → 139 out (120 reasoning)
be201d4
JSON Schema Structured Output
(raw) {"commit":"be201d4"}
openai/gpt-5.6-luna:high PASS 100% 191 in → 8 out (0 reasoning)
6ba999c
openai/gpt-5.6-luna:high__json_schema PASS 100% 234 in → 20 out (0 reasoning)
a8f4c8a
JSON Schema Structured Output
(raw) {"commit":"a8f4c8a"}
openai/gpt-5.6-luna:low PASS 100% 193 in → 7 out (0 reasoning)
aa759ac
openai/gpt-5.6-luna:low__json_schema PASS 100% 234 in → 16 out (0 reasoning)
bbb8519
JSON Schema Structured Output
(raw) {"commit":"bbb8519"}
openai/gpt-5.6-luna:max PASS 100% 201 in → 9 out (0 reasoning)
c208e1e
openai/gpt-5.6-luna:max__json_schema PASS 100% 228 in → 17 out (0 reasoning)
fa324e3
JSON Schema Structured Output
(raw) {"commit":"fa324e3"}
openai/gpt-5.6-luna:medium PASS 100% 195 in → 8 out (0 reasoning)
49125d0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 232 in → 18 out (0 reasoning)
b2b693d
JSON Schema Structured Output
(raw) {"commit":"b2b693d"}
openai/gpt-5.6-luna:none PASS 100% 193 in → 8 out (0 reasoning)
3a527ef
openai/gpt-5.6-luna:none__json_schema PASS 100% 234 in → 18 out (0 reasoning)
6dc5f89
JSON Schema Structured Output
(raw) {"commit":"6dc5f89"}
openai/gpt-5.6-luna:xhigh PASS 100% 195 in → 9 out (0 reasoning)
67e29b9
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 230 in → 17 out (0 reasoning)
70babd2
JSON Schema Structured Output
(raw) {"commit":"70babd2"}
openai/gpt-5.6-sol:high PASS 100% 197 in → 8 out (0 reasoning)
eef48f7
openai/gpt-5.6-sol:high__json_schema PASS 100% 228 in → 16 out (0 reasoning)
05782cb
JSON Schema Structured Output
(raw) {"commit":"05782cb"}
openai/gpt-5.6-sol:low PASS 100% 193 in → 10 out (0 reasoning)
c0e5e71
openai/gpt-5.6-sol:low__json_schema PASS 100% 232 in → 18 out (0 reasoning)
a1500f0
JSON Schema Structured Output
(raw) {"commit":"a1500f0"}
openai/gpt-5.6-sol:max PASS 100% 189 in → 8 out (0 reasoning)
0807a64
openai/gpt-5.6-sol:max__json_schema PASS 100% 234 in → 16 out (0 reasoning)
019e768
JSON Schema Structured Output
(raw) {"commit":"019e768"}
openai/gpt-5.6-sol:medium PASS 100% 193 in → 9 out (0 reasoning)
f7d012b
openai/gpt-5.6-sol:medium__json_schema PASS 100% 232 in → 17 out (0 reasoning)
78919a0
JSON Schema Structured Output
(raw) {"commit":"78919a0"}
openai/gpt-5.6-sol:none PASS 100% 191 in → 7 out (0 reasoning)
206af45
openai/gpt-5.6-sol:none__json_schema PASS 100% 226 in → 17 out (0 reasoning)
a6098dd
JSON Schema Structured Output
(raw) {"commit":"a6098dd"}
openai/gpt-5.6-sol:xhigh PASS 100% 187 in → 9 out (0 reasoning)
6ce0dc2
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 228 in → 17 out (0 reasoning)
3691c90
JSON Schema Structured Output
(raw) {"commit":"3691c90"}
openai/gpt-5.6-terra:high PASS 100% 197 in → 9 out (0 reasoning)
3742d4f
openai/gpt-5.6-terra:high__json_schema PASS 100% 234 in → 17 out (0 reasoning)
efa3b52
JSON Schema Structured Output
(raw) {"commit":"efa3b52"}
openai/gpt-5.6-terra:low PASS 100% 189 in → 9 out (0 reasoning)
7c5cb00
openai/gpt-5.6-terra:low__json_schema PASS 100% 232 in → 18 out (0 reasoning)
0ff31b9
JSON Schema Structured Output
(raw) {"commit":"0ff31b9"}
openai/gpt-5.6-terra:max PASS 100% 197 in → 46 out (34 reasoning)
9c9b11f
openai/gpt-5.6-terra:max__json_schema PASS 100% 238 in → 18 out (0 reasoning)
a8d99ce
JSON Schema Structured Output
(raw) {"commit":"a8d99ce"}
openai/gpt-5.6-terra:medium PASS 100% 191 in → 8 out (0 reasoning)
337fb1d
openai/gpt-5.6-terra:medium__json_schema PASS 100% 236 in → 16 out (0 reasoning)
6039328
JSON Schema Structured Output
(raw) {"commit":"6039328"}
openai/gpt-5.6-terra:none PASS 100% 189 in → 8 out (0 reasoning)
10012f5
openai/gpt-5.6-terra:none__json_schema PASS 100% 234 in → 18 out (0 reasoning)
0ddaa5e
JSON Schema Structured Output
(raw) {"commit":"0ddaa5e"}
openai/gpt-5.6-terra:xhigh PASS 100% 187 in → 7 out (0 reasoning)
faa593d
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 230 in → 17 out (0 reasoning)
c581d42
JSON Schema Structured Output
(raw) {"commit":"c581d42"}
openai/gpt-oss-120b:high PASS 100% 252 in → 175 out (175 reasoning)
150bd72 (reset config to zero)
openai/gpt-oss-120b:high__json_schema PASS 100% 256 in → 351 out (331 reasoning)
d9de52e
JSON Schema Structured Output
(raw) { "commit": "d9de52e" }
openai/gpt-oss-120b:low PASS 100% 243 in → 47 out (31 reasoning)
7a8c67a
openai/gpt-oss-120b:medium PASS 100% 254 in → 107 out (87 reasoning)
6c0cb78 – “reset config to zero”
openai/gpt-oss-120b:medium__json_schema PASS 100% 256 in → 115 out (103 reasoning)
b680daa
JSON Schema Structured Output
(raw) { "commit": "b680daa" }
openai/gpt-oss-120b:xhigh PASS 100% 256 in → 232 out (222 reasoning)
1c3cf15
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 248 in → 398 out (373 reasoning)
573a69b
JSON Schema Structured Output
(raw) { "commit": "573a69b" }
openai/gpt-oss-20b:high PASS 100% 246 in → 294 out (289 reasoning)
d24261a
openai/gpt-oss-20b:high__json_schema PASS 100% 250 in → 468 out (443 reasoning)
313a018
JSON Schema Structured Output
(raw) { "commit": "313a018" }
openai/gpt-oss-20b:low PASS 100% 254 in → 166 out (49 reasoning)
**Process** The tests pass for the first two commits (`aa36ec7` and `60437e4`). The next commit `64d3e1a` introduces a failure, and the subsequent commit `e40efa2` also fails. Since the failure appears at `64d3e1a`—the earliest point where the tests break—this is the commit that introduced the bug. **Answer** `64d3e1a`
openai/gpt-oss-20b:low__json_schema PASS 100% 248 in → 63 out (38 reasoning)
fc961a4
JSON Schema Structured Output
(raw) { "commit": "fc961a4" }
openai/gpt-oss-20b:medium PASS 100% 256 in → 133 out (129 reasoning)
bba2790
openai/gpt-oss-20b:medium__json_schema PASS 100% 258 in → 207 out (184 reasoning)
0a2e3ec
JSON Schema Structured Output
(raw) { "commit": "0a2e3ec" }
openai/gpt-oss-20b:xhigh PASS 100% 250 in → 299 out (289 reasoning)
d8cf008 (reset config to zero)
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 258 in → 12 out (0 reasoning)
72a2c96
JSON Schema Structured Output
(raw) {"commit": "72a2c96"}
poolside/laguna-m.1:high PASS 100% 220 in → 502 out (494 reasoning)
0dee434
poolside/laguna-m.1:high__json_schema PASS 100% 226 in → 799 out (785 reasoning)
5f8b74c
JSON Schema Structured Output
(raw) {"commit": "5f8b74c"}
poolside/laguna-m.1:low PASS 100% 222 in → 498 out (488 reasoning)
0d1b682
poolside/laguna-m.1:low__json_schema PASS 100% 230 in → 403 out (389 reasoning)
00a5f8b
JSON Schema Structured Output
(raw) {"commit": "00a5f8b"}
poolside/laguna-m.1:medium PASS 100% 222 in → 284 out (275 reasoning)
c97ab5d
poolside/laguna-m.1:medium__json_schema PASS 100% 228 in → 634 out (616 reasoning)
38e2359
JSON Schema Structured Output
(raw) { "commit": "38e2359" }
poolside/laguna-m.1:none PASS 100% 226 in → 202 out (0 reasoning)
The bad commit breaks the build, and using a binary search approach, we find that the commit "reset config to zero" (576633d) introduced the bug. Here's the step-by-step reasoning: 1. **Initial test results**: - `138ecbf` (Initial) → PASS - `b963371` (set config) → PASS - `576633d` (reset config) → FAIL - `b184c38` (final config) → FAIL 2. **Binary search midpoint**: Test a commit between `b963371` (last good) and `576633d` (first bad). There are no intermediate commits, so the first bad commit is **576633d**. **Answer**: 576633d
poolside/laguna-m.1:none__json_schema PASS 100% 226 in → 16 out (0 reasoning)
5835ed9
JSON Schema Structured Output
(raw) { "commit": "5835ed9" }
poolside/laguna-m.1:xhigh PASS 100% 224 in → 559 out (495 reasoning)
The bad commit is **bfb2d76**, as it is the first commit where the tests failed, following the last passing commit (b2d17e3). Subsequent commits either retained or exacerbated the issue but did not fix it. **Answer:** bfb2d76
poolside/laguna-m.1:xhigh__json_schema PASS 100% 222 in → 687 out (670 reasoning)
f22dc8c
JSON Schema Structured Output
(raw) { "commit": "f22dc8c" }
poolside/laguna-xs-2.1:high PASS 100% 220 in → 390 out (382 reasoning)
93e8f68
poolside/laguna-xs-2.1:high__json_schema PASS 100% 224 in → 281 out (269 reasoning)
dc4c7be
JSON Schema Structured Output
(raw) {"commit": "dc4c7be"}
poolside/laguna-xs-2.1:low PASS 100% 218 in → 365 out (300 reasoning)
The bad commit is the first one that introduced a test failure. Since the test results show that `5aeaa1f reset config to zero` is the first commit where the test fails (after passing for `4b7fcbe`), this is the commit that introduced the bug. 5aeaa1f
poolside/laguna-xs-2.1:low__json_schema PASS 100% 222 in → 629 out (616 reasoning)
ea25138
JSON Schema Structured Output
(raw) {"commit": "ea25138"}
poolside/laguna-xs-2.1:medium PASS 100% 218 in → 452 out (373 reasoning)
The bad commit is 6ddfebf, as it is the first commit in the history where the test results transition from passing (in 5ad4a10) to failing (6ddfebf). Subsequent commits (3e6106d) also fail, confirming that 6ddfebf introduced the bug. Answer: 6ddfebf
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 220 in → 263 out (245 reasoning)
3571476
JSON Schema Structured Output
(raw) { "commit": "3571476" }
poolside/laguna-xs-2.1:none PASS 100% 230 in → 7 out (0 reasoning)
a381a58
poolside/laguna-xs-2.1:none__json_schema PASS 100% 226 in → 18 out (0 reasoning)
0b80634
JSON Schema Structured Output
(raw) { "commit": "0b80634" }
poolside/laguna-xs-2.1:xhigh PASS 100% 226 in → 521 out (411 reasoning)
The bad commit is identified by tracing the first failure in the test results. Since the initial commits (a37a6c4 and b3f3a02) pass, but the next commit (60729ef) fails, the bug was introduced here. Subsequent failures (86ec811) are likely due to this initial breakage. **Bad commit hash:** `60729ef` **Or commit subject line:** `reset config to zero`
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 230 in → 326 out (312 reasoning)
8c73c5c
JSON Schema Structured Output
(raw) {"commit": "8c73c5c"}
poolside/laguna-xs.2:high PASS 100% 263 in → 368 out (261 reasoning)
Looking at the test results: - **d1eaa2f** (Initial commit): PASS - **d12fd7d** (set config value): PASS - **48f1e1b** (reset config to zero): FAIL - **45e519b** (final config): FAIL The first commit where tests fail is **48f1e1b** "reset config to zero". This is the commit that introduced the bug.
poolside/laguna-xs.2:high__json_schema PASS 100% 259 in → 237 out (224 reasoning)
c57a2fa
JSON Schema Structured Output
(raw) {"commit": "c57a2fa"}
poolside/laguna-xs.2:low PASS 100% 265 in → 313 out (201 reasoning)
Based on the test results, I can see that: 1. `38a634c Initial commit`: PASS 2. `f3f9aa2 set config value`: PASS 3. `2878663 reset config to zero`: FAIL ← First failure 4. `f36e986 final config`: FAIL The bad commit is **2878663** (reset config to zero), as this is the first commit where the tests started failing.
poolside/laguna-xs.2:low__json_schema PASS 100% 253 in → 287 out (274 reasoning)
dd30353
JSON Schema Structured Output
(raw) {"commit": "dd30353"}
poolside/laguna-xs.2:medium PASS 100% 261 in → 403 out (269 reasoning)
Based on the test results, I can see that: - Commit 765f868 (Initial commit): PASS - Commit 0966c20 (set config value): PASS - Commit 0975cbc (reset config to zero): FAIL - Commit 5bb3f32 (final config): FAIL The tests pass up through commit 0966c20, but fail starting with commit 0975cbc. This indicates that commit 0975cbc introduced the bug. The bad commit is: **0975cbc**
poolside/laguna-xs.2:medium__json_schema PASS 100% 255 in → 245 out (233 reasoning)
8cd9bf2
JSON Schema Structured Output
(raw) {"commit": "8cd9bf2"}
poolside/laguna-xs.2:none PASS 100% 267 in → 132 out (0 reasoning)
Looking at the test results by commit: 1. `32c4a18 Initial commit`: PASS 2. `1556f79 set config value`: PASS 3. `8e8724b reset config to zero`: FAIL 4. `1512923 final config`: FAIL The first commit where the tests failed is `8e8724b reset config to zero`. Since the tests were passing before this commit and then started failing, this is the commit that introduced the bug. The bad commit is: `8e8724b`
poolside/laguna-xs.2:none__json_schema PASS 100% 261 in → 13 out (0 reasoning)
239403a
JSON Schema Structured Output
(raw) {"commit": "239403a"}
poolside/laguna-xs.2:xhigh PASS 100% 261 in → 315 out (225 reasoning)
Based on the test results, I can see that: - Commit 01e9d75 (Initial commit): PASS - Commit 5e867ee (set config value): PASS - Commit e35eee3 (reset config to zero): FAIL The bad commit is **e35eee3** ("reset config to zero") - this is the first commit where tests started failing.
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 255 in → 298 out (285 reasoning)
4c03ad9
JSON Schema Structured Output
(raw) {"commit": "4c03ad9"}
qwen/qwen3.6-27b:high PASS 100% 222 in → 1,382 out (1,091 reasoning)
3646efb
qwen/qwen3.6-27b:high__json_schema PASS 100% 222 in → 493 out (401 reasoning)
06be1dd
JSON Schema Structured Output
(raw) { "commit": "06be1dd" }
qwen/qwen3.6-27b:low PASS 100% 226 in → 513 out (383 reasoning)
4318957
qwen/qwen3.6-27b:low__json_schema PASS 100% 218 in → 500 out (374 reasoning)
7861cb5
JSON Schema Structured Output
(raw) { "commit": "7861cb5" }
qwen/qwen3.6-27b:medium PASS 100% 218 in → 552 out (456 reasoning)
a49ce92
qwen/qwen3.6-27b:medium__json_schema PASS 100% 218 in → 455 out (1 reasoning)
9f2d0fa
JSON Schema Structured Output
(raw) { "commit": "9f2d0fa" }
qwen/qwen3.6-27b:none PASS 100% 226 in → 8 out (0 reasoning)
80e222e
qwen/qwen3.6-27b:none__json_schema PASS 100% 222 in → 17 out (0 reasoning)
fc2953c
JSON Schema Structured Output
(raw) { "commit": "fc2953c" }
qwen/qwen3.6-27b:xhigh PASS 100% 212 in → 1,027 out (831 reasoning)
11b9972
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 220 in → 841 out (637 reasoning)
a94764b
JSON Schema Structured Output
(raw) { "commit": "a94764b" }
qwen/qwen3.6-35b-a3b:high PASS 100% 222 in → 339 out (271 reasoning)
f44cbc3
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 220 in → 667 out (530 reasoning)
c81f357
JSON Schema Structured Output
(raw) { "commit": "c81f357" }
qwen/qwen3.6-35b-a3b:low PASS 100% 212 in → 309 out (268 reasoning)
1b7ec32
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 218 in → 359 out (287 reasoning)
ec61d67
JSON Schema Structured Output
(raw) { "commit": "ec61d67" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 216 in → 600 out (483 reasoning)
0e262be
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 222 in → 513 out (428 reasoning)
28cfe1c
JSON Schema Structured Output
(raw) { "commit": "28cfe1c" }
qwen/qwen3.6-35b-a3b:none PASS 100% 216 in → 116 out (0 reasoning)
The test results show that the build passed for `8acc0ca set config value`. However, the subsequent commit `c4ef091 reset config to zero` caused the build to fail (FAIL). Since `c4ef091` is the first commit in the history that introduced the failure, it is the bad commit. The next commit, `ad7a38d`, also fails, but it did not introduce the bug; it merely continued the broken state. Therefore, the bad commit is: c4ef091
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 222 in → 17 out (0 reasoning)
dc67185
JSON Schema Structured Output
(raw) { "commit": "dc67185" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 214 in → 580 out (480 reasoning)
da88ab2
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 222 in → 399 out (294 reasoning)
696f1a9
JSON Schema Structured Output
(raw) { "commit": "696f1a9" }
qwen/qwen3.6-flash:high PASS 100% 222 in → 429 out (417 reasoning)
b5b52f5
qwen/qwen3.6-flash:low PASS 100% 220 in → 299 out (289 reasoning)
48b4acf
qwen/qwen3.6-flash:medium PASS 100% 220 in → 592 out (580 reasoning)
346036f
qwen/qwen3.6-flash:none PASS 100% 222 in → 7 out (0 reasoning)
70b4886
qwen/qwen3.6-flash:xhigh PASS 100% 224 in → 369 out (357 reasoning)
9445031
qwen/qwen3.7-flash:high PASS 100% 220 in → 316 out (304 reasoning)
3887274
qwen/qwen3.7-flash:low PASS 100% 224 in → 361 out (350 reasoning)
71cf4e6
qwen/qwen3.7-flash:max PASS 100% 218 in → 380 out (369 reasoning)
4e506fd
qwen/qwen3.7-flash:medium PASS 100% 222 in → 607 out (596 reasoning)
2257edf
qwen/qwen3.7-flash:none PASS 100% 218 in → 4 out (0 reasoning)
4bab5ab
qwen/qwen3.7-flash:xhigh PASS 100% 212 in → 1,003 out (992 reasoning)
5df576b
qwen/qwen3.7-max:high PASS 100% 210 in → 284 out (194 reasoning)
The bad commit is **9bff8ef** ("reset config to zero"). **Reasoning:** The test results show that the build was passing through commit `c71e1ef` and first failed at commit `9bff8ef`. This is the commit that introduced the bug. The subsequent commit (`2ddd247`) also fails because it inherits the broken state from `9bff8ef`.
qwen/qwen3.7-max:low PASS 100% 220 in → 275 out (137 reasoning)
The bad commit is **a0548b8** ("reset config to zero"). **Reasoning:** - `091d156` (Initial commit): PASS ✅ - `9bdc9ce` (set config value): PASS ✅ - `a0548b8` (reset config to zero): **FAIL** ❌ ← Bug introduced here - `3d4ce5b` (final config): FAIL ❌ (still broken) The build transitioned from PASS to FAIL at commit `a0548b8`, making it the commit that introduced the bug.
qwen/qwen3.7-max:medium PASS 100% 226 in → 195 out (183 reasoning)
7c24498
qwen/qwen3.7-max:none PASS 100% 224 in → 56 out (0 reasoning)
Based on the test results, the build was passing at commit `3d992e1` and started failing at commit `e94056c`. Therefore, the bad commit that introduced the bug is: **e94056c**
qwen/qwen3.7-max:xhigh PASS 100% 218 in → 192 out (180 reasoning)
552d412
qwen/qwen3.7-plus:high PASS 100% 224 in → 276 out (140 reasoning)
The bad commit is **73a6078** ("reset config to zero"). **Reasoning:** Looking at the test results chronologically: - `1889f4b` Initial commit → PASS - `3528345` set config value → PASS - `73a6078` reset config to zero → **FAIL** (first failure) - `3bca3d0` final config → FAIL The first commit that causes the build to fail is `73a6078`, which means this is the commit that introduced the bug.
qwen/qwen3.7-plus:low PASS 100% 222 in → 394 out (236 reasoning)
The bad commit is **`832e43e`** ("reset config to zero"). Here's the reasoning: - `563ce9e` (Initial commit): ✅ PASS - `6b8c5b5` (set config value): ✅ PASS - **`832e43e` (reset config to zero): ❌ FAIL** ← First failure - `fae8604` (final config): ❌ FAIL The build was passing at `6b8c5b5` and started failing at `832e43e`. Therefore, commit **`832e43e`** is the one that introduced the bug.
qwen/qwen3.7-plus:medium PASS 100% 216 in → 310 out (173 reasoning)
The bad commit is **87940f3**. **Reasoning:** Looking at the test results in chronological order: - `ca5e6db` (Initial commit): ✅ PASS - `37e74e7` (set config value): ✅ PASS - `87940f3` (reset config to zero): ❌ FAIL ← first failure - `8acd1de` (final config): ❌ FAIL The transition from PASS to FAIL happens at commit `87940f3` ("reset config to zero"), making it the commit that introduced the bug.
qwen/qwen3.7-plus:none PASS 100% 222 in → 5 out (0 reasoning)
50aebcd
qwen/qwen3.7-plus:xhigh PASS 100% 222 in → 348 out (196 reasoning)
The bad commit is **605b25d** ("reset config to zero"). **Reasoning:** - `75ac086` Initial commit → PASS ✅ - `d1514dc` set config value → PASS ✅ - `605b25d` reset config to zero → **FAIL ❌** (first failure) - `c76d62a` final config → FAIL ❌ The last good commit was `d1514dc`. The very next commit, `605b25d` ("reset config to zero"), is where the build first breaks. This is the commit that introduced the bug.
tencent/hy3:high PASS 100% 193 in → 527 out (521 reasoning)
24db72e
tencent/hy3:high__json_schema PASS 100% 191 in → 949 out (906 reasoning)
5b42e76
JSON Schema Structured Output
(raw) { "commit": "5b42e76" }
tencent/hy3:low PASS 100% 193 in → 386 out (380 reasoning)
4614e02
tencent/hy3:low__json_schema PASS 100% 201 in → 593 out (528 reasoning)
2551481
JSON Schema Structured Output
(raw) { "commit": "2551481" }
tencent/hy3:medium PASS 100% 197 in → 446 out (440 reasoning)
69bb8af
tencent/hy3:medium__json_schema PASS 100% 195 in → 305 out (250 reasoning)
bff800b
JSON Schema Structured Output
(raw) { "commit": "bff800b" }
tencent/hy3:none PASS 100% 202 in → 6 out (0 reasoning)
dcf9f82
tencent/hy3:none__json_schema PASS 100% 206 in → 14 out (0 reasoning)
e18f3f6
JSON Schema Structured Output
(raw) { "commit": "e18f3f6" }
tencent/hy3:xhigh PASS 100% 197 in → 544 out (538 reasoning)
476b9ac
tencent/hy3:xhigh__json_schema PASS 100% 197 in → 619 out (589 reasoning)
53fac11
JSON Schema Structured Output
(raw) { "commit": "53fac11" }
thinkingmachines/inkling-small:high PASS 100% 202 in → 124 out (92 reasoning)
4614320
thinkingmachines/inkling-small:high__json_schema PASS 100% 200 in → 36 out (21 reasoning)
19694ac
JSON Schema Structured Output
(raw) { "commit": "19694ac" }
thinkingmachines/inkling-small:low PASS 100% 196 in → 24 out (15 reasoning)
8271807
thinkingmachines/inkling-small:low__json_schema PASS 100% 208 in → 34 out (18 reasoning)
7837e39
JSON Schema Structured Output
(raw) { "commit": "7837e39" }
thinkingmachines/inkling-small:max PASS 100% 196 in → 30 out (21 reasoning)
684cb84
thinkingmachines/inkling-small:max__json_schema PASS 100% 200 in → 128 out (111 reasoning)
d76c45d
JSON Schema Structured Output
(raw) { "commit": "d76c45d" }
thinkingmachines/inkling-small:medium PASS 100% 198 in → 114 out (80 reasoning)
9a94726
thinkingmachines/inkling-small:medium__json_schema PASS 100% 204 in → 127 out (109 reasoning)
e8bf9f8
JSON Schema Structured Output
(raw) { "commit": "e8bf9f8" }
thinkingmachines/inkling-small:none PASS 100% 196 in → 7 out (0 reasoning)
b9bea33<|end_message|>
thinkingmachines/inkling-small:xhigh PASS 100% 206 in → 42 out (31 reasoning)
4ad1ca1
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 200 in → 39 out (22 reasoning)
ce1306b
JSON Schema Structured Output
(raw) { "commit": "ce1306b" }
thinkingmachines/inkling:high PASS 100% 206 in → 121 out (88 reasoning)
62cddb5
thinkingmachines/inkling:low PASS 100% 196 in → 20 out (10 reasoning)
624b889
thinkingmachines/inkling:low__json_schema PASS 100% 200 in → 34 out (16 reasoning)
a65fbe0
JSON Schema Structured Output
(raw) { "commit": "a65fbe0" }
thinkingmachines/inkling:max PASS 100% 200 in → 115 out (106 reasoning)
e918777
thinkingmachines/inkling:max__json_schema PASS 100% 204 in → 131 out (114 reasoning)
5a4ce91
JSON Schema Structured Output
(raw) { "commit": "5a4ce91" }
thinkingmachines/inkling:medium PASS 100% 196 in → 75 out (65 reasoning)
fff038a
thinkingmachines/inkling:none PASS 100% 202 in → 9 out (0 reasoning)
6af3b72
thinkingmachines/inkling:xhigh PASS 100% 202 in → 223 out (184 reasoning)
The build first fails at `ec8de77`, so that is the bad commit. **ec8de77** (or "reset config to zero")
x-ai/grok-4.3:high PASS 100% 370 in → 416 out (412 reasoning)
36a377c
x-ai/grok-4.3:high__json_schema PASS 100% 427 in → 400 out (391 reasoning)
d56d0fd
JSON Schema Structured Output
(raw) {"commit":"d56d0fd"}
x-ai/grok-4.3:low PASS 100% 374 in → 343 out (238 reasoning)
dd56d20 The tests show that commits 40ec770 and 4627df2 both exit with code 0 (PASS). The very next commit in the linear history, dd56d20, is the first to exit with code 1 (FAIL). Therefore dd56d20 is the commit that introduced the regression; the subsequent commit 0e58d05 merely inherits the same failure. This is the classic “first failing commit” result produced by a binary-search or linear walk over the history.
x-ai/grok-4.3:low__json_schema PASS 100% 437 in → 270 out (262 reasoning)
3bdb18f
JSON Schema Structured Output
(raw) {"commit":"3bdb18f"}
x-ai/grok-4.3:max PASS 100% 364 in → 456 out (453 reasoning)
efb9dab
x-ai/grok-4.3:max__json_schema PASS 100% 429 in → 409 out (401 reasoning)
9dbaddf
JSON Schema Structured Output
(raw) {"commit":"9dbaddf"}
x-ai/grok-4.3:medium PASS 100% 374 in → 488 out (398 reasoning)
2479e02 The test results show a clean transition from PASS to FAIL between commits 721a468 and 2479e02; therefore 2479e02 is the first commit whose tree produces a failing build and is the commit that introduced the bug. Later commits remain failing, which is consistent with the bug having already been present. The remaining commits are either known-good (before the failure) or are irrelevant (after the failure).
x-ai/grok-4.3:medium__json_schema PASS 100% 437 in → 400 out (391 reasoning)
fd800b4
JSON Schema Structured Output
(raw) {"commit": "fd800b4"}
x-ai/grok-4.3:none PASS 100% 366 in → 4 out (0 reasoning)
72910d7
x-ai/grok-4.3:none__json_schema PASS 100% 435 in → 9 out (0 reasoning)
3e3c881
JSON Schema Structured Output
(raw) {"commit":"3e3c881"}
x-ai/grok-4.3:xhigh PASS 100% 370 in → 388 out (383 reasoning)
f3e45ec
x-ai/grok-4.3:xhigh__json_schema PASS 100% 435 in → 459 out (451 reasoning)
2d83374
JSON Schema Structured Output
(raw) {"commit":"2d83374"}
x-ai/grok-4.5:high PASS 100% 392 in → 220 out (217 reasoning)
442d153
x-ai/grok-4.5:high__json_schema PASS 100% 461 in → 266 out (257 reasoning)
5df5686
JSON Schema Structured Output
(raw) {"commit":"5df5686"}
x-ai/grok-4.5:low PASS 100% 390 in → 218 out (214 reasoning)
008be1d
x-ai/grok-4.5:low__json_schema PASS 100% 463 in → 196 out (186 reasoning)
5208a8a
JSON Schema Structured Output
(raw) {"commit":"5208a8a"}
x-ai/grok-4.5:max PASS 100% 400 in → 355 out (350 reasoning)
3177b0c
x-ai/grok-4.5:max__json_schema PASS 100% 469 in → 224 out (212 reasoning)
fb859b4
JSON Schema Structured Output
(raw) { "commit": "fb859b4" }
x-ai/grok-4.5:medium PASS 100% 388 in → 214 out (211 reasoning)
ab56180
x-ai/grok-4.5:medium__json_schema PASS 100% 461 in → 218 out (209 reasoning)
b4615df
JSON Schema Structured Output
(raw) {"commit":"b4615df"}
x-ai/grok-4.5:xhigh PASS 100% 400 in → 245 out (239 reasoning)
f6a79c4
x-ai/grok-4.5:xhigh__json_schema PASS 100% 463 in → 323 out (310 reasoning)
1b6fc86
JSON Schema Structured Output
(raw) { "commit": "1b6fc86" }
z-ai/glm-4.7-flash:high PASS 100% 194 in → 694 out (600 reasoning)
ec28f77
z-ai/glm-4.7-flash:low PASS 100% 202 in → 595 out (480 reasoning)
8bd1e4e
z-ai/glm-4.7-flash:medium PASS 100% 196 in → 644 out (637 reasoning)
210a9b2
z-ai/glm-4.7-flash:none PASS 100% 192 in → 6 out (0 reasoning)
873b545
z-ai/glm-4.7-flash:none__json_schema PASS 100% 196 in → 13 out (0 reasoning)
07126e4
JSON Schema Structured Output
(raw) { "commit": "07126e4" }
z-ai/glm-4.7-flash:xhigh PASS 100% 196 in → 727 out (720 reasoning)
26215d6
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 192 in → 451 out (391 reasoning)
37764f4
JSON Schema Structured Output
(raw) { "commit": "37764f4" }
z-ai/glm-5.2:high PASS 100% 203 in → 159 out (152 reasoning)
df2b804
z-ai/glm-5.2:low PASS 100% 199 in → 143 out (137 reasoning)
49cd355
z-ai/glm-5.2:low__json_schema PASS 100% 211 in → 125 out (109 reasoning)
536a1f2
JSON Schema Structured Output
(raw) { "commit": "536a1f2" }
z-ai/glm-5.2:medium PASS 100% 203 in → 137 out (131 reasoning)
bb2bdff
z-ai/glm-5.2:none PASS 100% 195 in → 4 out (0 reasoning)
31fe377
z-ai/glm-5.2:none__json_schema PASS 100% 197 in → 12 out (0 reasoning)
ad803be
JSON Schema Structured Output
(raw) { "commit": "ad803be" }
z-ai/glm-5.2:xhigh PASS 100% 205 in → 137 out (122 reasoning)
c82194e
z-ai/glm-5.2:xhigh__json_schema PASS 100% 201 in → 239 out (208 reasoning)
a622699
JSON Schema Structured Output
(raw) { "commit": "a622699" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 4ee3001
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: 56d9cc5
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Invalid JSON. Output: The **bad commit** is: **c92d56f** **Reason:** - The initial commit (`c7d8bf5`) passed. - After `c92d56f`, the build still passed, so it's not this one. - The next commit (`1c0d4a0`) failed, but we need to find the *first* commit that caused the build to break from passing to failing. However, since both `c92d56f` and `1c0d4a0` lead to failure, and the problem states there is **one bad commit**, we look for the commit whose change directly caused the build to fail when applied after all previous commits. But let's check the sequence: - Pass → Pass → **Pass** → **FAIL** (after c92d56f) - Then FAIL again after c1c0d4a0 Since the build fails at `1c0d4a0`, but also fails at `c92d56f`, and the problem says only **one bad commit**, it means the commit that actually broke the build must be the one that changed something that caused the failure. Given the test outputs, the **first commit that causes a failure** after a passing state is **c92d56f**. However, if the build fails at `1c0d4a0`, but `c92d56f` passed, then `c92d56f` is not the culprit. But the problem says "one bad commit", so likely the commit that introduced the bug is the one that made the build fail **from a passing state**. If `c92d56f` passes, then the next commit (`1c0d4a0`) fails, so that would be the bad commit. **Final Answer:** `1c0d4a0` --- **Summary Table:** | Commit Hash | Result | |-------------|----------| | 11c5df2 | FAIL | | c92d56f | PASS | | 1c0d4a0 | FAIL | | c7d8bf5 | PASS | So the **bad commit** is: ``` 1c0d4a0 ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: bff4eb1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 224 in → 8 out (0 reasoning)
c444b3d
Failure: Expected target hash/prefix or subject 'reset config to zero'
openai/gpt-oss-120b:low__json_schema FAIL 0% 254 in → 11 out (0 reasoning)
d7600
JSON Schema Structured Output
(raw) { "commit": "d7600" }
Failure: Expected target hash/prefix or subject 'reset config to zero'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2fd3fe6f-56c7-9341-9907-b7d5dba294e7","request_id":"2fd3fe6f-56c7-9341-9907-b7d5dba294e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-20fa813e-1ee4-93ca-ae40-693d9cbaa2f0","request_id":"20fa813e-1ee4-93ca-ae40-693d9cbaa2f0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-086d9249-93c7-9910-99cc-62eeef9458a0","request_id":"086d9249-93c7-9910-99cc-62eeef9458a0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c9aca390-bfb2-9878-af25-e01ec6385b65","request_id":"c9aca390-bfb2-9878-af25-e01ec6385b65"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b33e298b-3eac-9ff4-98d3-2b37e5c0ce73","request_id":"b33e298b-3eac-9ff4-98d3-2b37e5c0ce73"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4952e304-c385-9b4d-888f-d97fd2f918d6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-80ec52de-0c62-9fda-a31d-a3ac26d43563"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0a10970b-3016-9ca7-af2b-d3e5ee80f045"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-36b190f3-672f-9af6-9dfd-c50a75dd3ba9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d8d16608-ec96-9aac-9a66-01cb97256b19"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e0cf5572-ec58-9b83-ab6e-ea3b624568c9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d75edfe2-4c16-975c-bab8-6090a2b2f185","request_id":"d75edfe2-4c16-975c-bab8-6090a2b2f185"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cf4ae14b-cacc-9880-b4ca-cd727f346ef0","request_id":"cf4ae14b-cacc-9880-b4ca-cd727f346ef0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c7cf6625-2197-9ff5-bb2f-7dc16eefd2de","request_id":"c7cf6625-2197-9ff5-bb2f-7dc16eefd2de"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d33f039a-7455-9fa6-a335-2d930034223a","request_id":"d33f039a-7455-9fa6-a335-2d930034223a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90ebb05f-3203-9fdd-9e10-04c520819943","request_id":"90ebb05f-3203-9fdd-9e10-04c520819943"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-16b22307-4df5-9afb-87f3-469228e6677b","request_id":"16b22307-4df5-9afb-87f3-469228e6677b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-12fd7189-a13f-99dd-8411-2344e1f23066","request_id":"12fd7189-a13f-99dd-8411-2344e1f23066"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-657b59e0-2fa8-98c5-b45b-fdd7c5f0751b","request_id":"657b59e0-2fa8-98c5-b45b-fdd7c5f0751b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c152275c-a3de-9f14-88ca-07f38a77803d","request_id":"c152275c-a3de-9f14-88ca-07f38a77803d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6eae4724-232e-9d12-8b13-7c3475771552","request_id":"6eae4724-232e-9d12-8b13-7c3475771552"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: 5ac2172{"commit": "5ac2172"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: The first commit that fails is **2305e54** (subject: "reset config to zero"). So the bad commit is: **2305e54**
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: de750e5
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 4e27cb4{"commit": "4e27cb4"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)
Invalid JSON. Output: The build first fails at commit **ff82f02** ("reset config to zero"), so that’s the bad commit. Answer: `ff82f02` (or "reset config to zero")
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: f1d49af
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 549c61a
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 4 (char 3)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 4 (char 3)
Invalid JSON. Output: 9472cd5
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 5 (char 4)