Linear history with bad commit third from beginning - model identifies third commit
Tests ability to identify the third commit as bad. Evaluates bisect with the bug early but not at the very start.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '#!/bin/bash' > test.sh
  5. 05 echo '! grep -q "port: 8080" config/*.yaml 2>/dev/null' >> test.sh
  6. 06 chmod +x test.sh
  7. 07 git add test.sh
  8. 08 git commit -m 'Initial commit'
  9. 09 mkdir -p config && echo 'server: port: 3000' > config/default.yaml
  10. 10 git add config/default.yaml
  11. 11 git commit -m 'add default config'
  12. 12 echo 'server: port: 8080' > config/default.yaml
  13. 13 git add config/default.yaml
  14. 14 git commit -m 'change port'
  15. 15 echo 'server: port: 8080 feature: true' > config/default.yaml
  16. 16 git add config/default.yaml
  17. 17 git commit -m 'add feature flag'
  18. 18 mkdir -p src && echo 'const PORT = 3000;' > src/index.js
  19. 19 git add src/index.js
  20. 20 git commit -m 'add port constant'
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad. Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
change port

Scoped model quality, cost, API time, and token usage for git_bisect/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 295 in → 193 out (15 reasoning)
**Bad commit: `f470ad5` — "change port"** The bisection logic is straightforward: - `b773631` (Initial commit): PASS ✅ - `4decdd6` (add default config): PASS ✅ - `f470ad5` (change port): **FAIL** ❌ ← first failing commit - `08dd536` and `124a846` fail because they build on top of the broken commit The transition from PASS to FAIL occurs at **f470ad5 "change port"**, making it the commit that introduced the bug.
anthropic/claude-fable-5:high__json_schema PASS 100% 510 in → 13 out (0 reasoning)
e522775
JSON Schema Structured Output
(raw) {"commit": "e522775"}
anthropic/claude-fable-5:low PASS 100% 307 in → 36 out (0 reasoning)
fdd31c7 — "change port" is the bad commit (first failing commit after passing 50e3c78).
anthropic/claude-fable-5:low__json_schema PASS 100% 524 in → 15 out (0 reasoning)
92d9e55
JSON Schema Structured Output
(raw) {"commit": "92d9e55"}
anthropic/claude-fable-5:max PASS 100% 303 in → 425 out (42 reasoning)
**Bad commit: `52207d6` — "change port"** **Reasoning (bisect logic):** Walking through the test results in chronological order: | Commit | Subject | Result | |--------|---------|--------| | 29d9f37 | Initial commit | ✅ PASS | | 1d19aca | add default config | ✅ PASS | | **52207d6** | **change port** | ❌ **FAIL** ← first failure | | 18f8243 | add feature flag | ❌ FAIL | | fd6bfb9 | add port constant | ❌ FAIL | The build passed through `1d19aca` and first failed at `52207d6`. All subsequent commits fail because they build on top of the broken state. Therefore, the commit that introduced the bug is **`52207d6` ("change port")** — the transition point from PASS to FAIL.
anthropic/claude-fable-5:max__json_schema PASS 100% 518 in → 128 out (39 reasoning)
06b4c76
JSON Schema Structured Output
(raw) {"commit": "06b4c76"}
anthropic/claude-fable-5:medium PASS 100% 301 in → 57 out (0 reasoning)
The bad commit is **3c0ca42** ("change port"). Tests pass through 977447f, then first fail at 3c0ca42, so that commit introduced the bug.
anthropic/claude-fable-5:medium__json_schema PASS 100% 512 in → 14 out (0 reasoning)
03c2548
JSON Schema Structured Output
(raw) {"commit": "03c2548"}
anthropic/claude-fable-5:xhigh PASS 100% 297 in → 244 out (15 reasoning)
**Bad commit: `8743a9c` — "change port"** Reasoning (bisect-style analysis): - `1c6d822` (Initial commit): PASS ✅ - `a9527e6` (add default config): PASS ✅ - `8743a9c` (change port): **FAIL ❌ ← first failing commit** - `b405076` (add feature flag): FAIL (inherited breakage) - `7068f52` (add port constant): FAIL (inherited breakage) The build transitions from PASS to FAIL at commit `8743a9c` ("change port"), making it the commit that introduced the bug. The subsequent failures are just downstream of that broken commit.
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 520 in → 16 out (0 reasoning)
45a7cb5
JSON Schema Structured Output
(raw) {"commit": "45a7cb5"}
anthropic/claude-haiku-4.5:high PASS 100% 261 in → 463 out (255 reasoning)
# Analysis Looking at the test results chronologically: - **245e3ea** (Initial commit): PASS ✓ - **7811356** (add default config): PASS ✓ - **ec9b625** (change port): FAIL ✗ ← First failure - **c712735** (add feature flag): FAIL ✗ - **fba7d17** (add port constant): FAIL ✗ The build passes through the first two commits, then fails starting with **ec9b625**. All subsequent commits also fail, but the root cause is introduced in the first failing commit. **Answer: `ec9b625` (or "change port")**
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 441 in → 396 out (305 reasoning)
3a3c3de
JSON Schema Structured Output
(raw) {"commit": "3a3c3de"}
anthropic/claude-haiku-4.5:low PASS 100% 277 in → 465 out (208 reasoning)
# Answer **7ec2e18** (change port) ## Reasoning Looking at the test results in chronological order: - d831f40 (Initial commit): PASS ✓ - f8f7c44 (add default config): PASS ✓ - **7ec2e18 (change port): FAIL ✗** ← First failure - 48dd8c0 (add feature flag): FAIL - f1d6bf2 (add port constant): FAIL The commit **7ec2e18** is the first commit where the build breaks. All commits before it pass, and all commits after it (which build on top of it) also fail. This indicates that the bug was introduced in the "change port" commit.
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 437 in → 219 out (182 reasoning)
bedbef4
JSON Schema Structured Output
(raw) {"commit": "bedbef4"}
anthropic/claude-haiku-4.5:medium PASS 100% 267 in → 408 out (202 reasoning)
# Analysis Looking at the test results in chronological order: 1. **c678aa5** (Initial commit): ✅ PASS 2. **184d4da** (add default config): ✅ PASS 3. **37b28a0** (change port): ❌ FAIL ← Tests start failing here 4. **57b68b1** (add feature flag): ❌ FAIL 5. **d68e331** (add port constant): ❌ FAIL The first commit where tests failed is **37b28a0**, immediately after the last passing commit. **Answer: 37b28a0** (or "change port")
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 437 in → 219 out (162 reasoning)
a80eb8a
JSON Schema Structured Output
(raw) {"commit": "a80eb8a"}
anthropic/claude-haiku-4.5:none PASS 100% 240 in → 62 out (0 reasoning)
The bad commit is **f2c7a45** (change port). This is the first commit where the tests start failing. The commit "change port" introduced the bug that broke the build, and all subsequent commits also fail, indicating the bug was never fixed in later commits.
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 407 in → 12 out (0 reasoning)
46e91de
JSON Schema Structured Output
(raw) {"commit": "46e91de"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 269 in → 544 out (281 reasoning)
# Analysis Looking at the test results in chronological order: - **931785b** (Initial commit): PASS ✓ - **4cb8ea7** (add default config): PASS ✓ - **8cd75c3** (change port): **FAIL** ✗ ← First failure - **dc870e6** (add feature flag): FAIL ✗ - **b8e9d28** (add port constant): FAIL ✗ The build was passing up to commit `4cb8ea7`, and the first commit where tests start failing is `8cd75c3`. This is where the bug was introduced. **Answer: `8cd75c3` or `change port`**
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 439 in → 244 out (187 reasoning)
f9fcce5
JSON Schema Structured Output
(raw) {"commit": "f9fcce5"}
anthropic/claude-opus-4.6:high PASS 100% 234 in → 34 out (0 reasoning)
The bad commit is **28e6084** (`change port`). It is the first commit where the test fails, meaning it introduced the bug.
anthropic/claude-opus-4.6:high__json_schema PASS 100% 400 in → 12 out (0 reasoning)
36acbf4
JSON Schema Structured Output
(raw) {"commit":"36acbf4"}
anthropic/claude-opus-4.6:low PASS 100% 232 in → 9 out (0 reasoning)
02d88b2
anthropic/claude-opus-4.6:low__json_schema PASS 100% 404 in → 11 out (0 reasoning)
b96fc77
JSON Schema Structured Output
(raw) {"commit":"b96fc77"}
anthropic/claude-opus-4.6:max PASS 100% 234 in → 257 out (111 reasoning)
**Bad commit: `1f72311` ("change port")** **Reasoning:** The test results show that all commits up to and including `7b5cf24` (add default config) pass, but starting from `1f72311` (change port), all subsequent commits fail. This means `1f72311` is the first commit that introduced the failure, making it the bad commit. The later failures (`369d375`, `9e293fe`) are simply inheriting the bug introduced by `1f72311`.
anthropic/claude-opus-4.6:max__json_schema PASS 100% 408 in → 138 out (93 reasoning)
b56aec0
JSON Schema Structured Output
(raw) {"commit":"b56aec0"}
anthropic/claude-opus-4.6:medium PASS 100% 240 in → 10 out (0 reasoning)
**d0425b7**
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 410 in → 12 out (0 reasoning)
4cba54f
JSON Schema Structured Output
(raw) {"commit":"4cba54f"}
anthropic/claude-opus-4.6:none PASS 100% 238 in → 34 out (0 reasoning)
The bad commit is `90e1a30` ("change port"). It is the first commit where the test fails after a series of passing commits.
anthropic/claude-opus-4.6:none__json_schema PASS 100% 410 in → 12 out (0 reasoning)
36a9cf8
JSON Schema Structured Output
(raw) {"commit":"36a9cf8"}
anthropic/claude-opus-4.6:xhigh PASS 100% 232 in → 35 out (0 reasoning)
The bad commit is **3119f58** (`change port`). This is the first commit where the tests start failing, meaning it introduced the bug.
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 404 in → 12 out (0 reasoning)
e7fa93d
JSON Schema Structured Output
(raw) {"commit":"e7fa93d"}
anthropic/claude-opus-4.7:high PASS 100% 304 in → 11 out (0 reasoning)
94a2d9d
anthropic/claude-opus-4.7:high__json_schema PASS 100% 527 in → 16 out (0 reasoning)
02a4112
JSON Schema Structured Output
(raw) {"commit":"02a4112"}
anthropic/claude-opus-4.7:low PASS 100% 296 in → 10 out (0 reasoning)
c13f8fc
anthropic/claude-opus-4.7:low__json_schema PASS 100% 523 in → 16 out (0 reasoning)
851ebde
JSON Schema Structured Output
(raw) {"commit":"851ebde"}
anthropic/claude-opus-4.7:max PASS 100% 298 in → 330 out (36 reasoning)
Looking at the test results chronologically: - `a5ea606` Initial commit: **PASS** - `ecadd69` add default config: **PASS** - `493936e` change port: **FAIL** ← First failing commit - `f82070d` add feature flag: FAIL - `db116a1` add port constant: FAIL The transition from PASS to FAIL occurs at commit `493936e` "change port". All subsequent commits inherit the broken state. **Bad commit: `493936e` (change port)**
anthropic/claude-opus-4.7:max__json_schema PASS 100% 523 in → 17 out (0 reasoning)
92f75b1
JSON Schema Structured Output
(raw) {"commit":"92f75b1"}
anthropic/claude-opus-4.7:medium PASS 100% 306 in → 8 out (0 reasoning)
c14e225
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 523 in → 16 out (0 reasoning)
58546e8
JSON Schema Structured Output
(raw) {"commit":"58546e8"}
anthropic/claude-opus-4.7:none PASS 100% 302 in → 9 out (0 reasoning)
39ea660
anthropic/claude-opus-4.7:none__json_schema PASS 100% 515 in → 15 out (0 reasoning)
955744e
JSON Schema Structured Output
(raw) {"commit":"955744e"}
anthropic/claude-opus-4.7:xhigh PASS 100% 300 in → 10 out (0 reasoning)
4b87f24
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 529 in → 18 out (0 reasoning)
4c09f0a
JSON Schema Structured Output
(raw) {"commit":"4c09f0a"}
anthropic/claude-opus-4.8:high PASS 100% 303 in → 158 out (18 reasoning)
The bad commit is **`54e2ca6`** ("change port"). The test results show a clear transition from PASS to FAIL: - `d9ade0b` — PASS ✅ - `54e2ca6` — **FAIL** ❌ (first failing commit) - All subsequent commits remain FAIL Since `54e2ca6` is the first commit where the build broke (and everything before it passed), it introduced the bug.
anthropic/claude-opus-4.8:high__json_schema PASS 100% 522 in → 17 out (0 reasoning)
a4c8fa9
JSON Schema Structured Output
(raw) {"commit": "a4c8fa9"}
anthropic/claude-opus-4.8:low PASS 100% 299 in → 8 out (0 reasoning)
bc7d45b
anthropic/claude-opus-4.8:low__json_schema PASS 100% 516 in → 14 out (0 reasoning)
fba3844
JSON Schema Structured Output
(raw) {"commit":"fba3844"}
anthropic/claude-opus-4.8:max PASS 100% 297 in → 601 out (45 reasoning)
**Bad commit: `e99e829` ("change port")** ## Reasoning Looking at the test results in chronological order (oldest to newest), I can find the transition point from passing to failing: | Commit | Subject | Result | |--------|---------|--------| | 46f7337 | Initial commit | ✅ PASS | | c1e3c4e | add default config | ✅ PASS | | **e99e829** | **change port** | ❌ **FAIL** ← first failure | | 417e7a1 | add feature flag | ❌ FAIL | | 726269e | add port constant | ❌ FAIL | The build was healthy through `c1e3c4e`, then broke starting at `e99e829`. Since every commit **before** `e99e829` passes and every commit **from** `e99e829` onward fails, this is the commit that introduced the bug. The two later failures (`417e7a1` and `726269e`) are just inheriting the already-broken state — they aren't the root cause. This is exactly the transition point `git bisect` would converge on: ``` git bisect start git bisect bad 726269e # newest, known bad git bisect good c1e3c4e # known good # → bisect would identify e99e829 as the first bad commit ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 518 in → 62 out (39 reasoning)
6212044
JSON Schema Structured Output
(raw) {"commit": "6212044"}
anthropic/claude-opus-4.8:medium PASS 100% 313 in → 120 out (12 reasoning)
**8cf4ce6** ("change port") This is the bad commit. The build passes at `01283e1` and fails starting at `8cf4ce6`, with all subsequent commits inheriting the failure. Since `8cf4ce6` is the first commit to show FAIL (exit code 1), it introduced the bug.
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 520 in → 16 out (0 reasoning)
51f6a3d
JSON Schema Structured Output
(raw) {"commit": "51f6a3d"}
anthropic/claude-opus-4.8:none PASS 100% 289 in → 101 out (0 reasoning)
The bad commit is **`07945c2`** (`change port`). This is the first commit where the tests transition from PASS to FAIL. The commit immediately before it (`237b285 add default config`) passes, and every commit from `07945c2` onward fails, indicating this commit introduced the bug that broke the build.
anthropic/claude-opus-4.8:none__json_schema PASS 100% 516 in → 13 out (0 reasoning)
71c81de
JSON Schema Structured Output
(raw) {"commit": "71c81de"}
anthropic/claude-opus-4.8:xhigh PASS 100% 305 in → 366 out (40 reasoning)
**Bad commit: `63e0c22` ("change port")** **Reasoning:** Looking at the test results in chronological order (oldest → newest): | Commit | Subject | Result | |--------|---------|--------| | d842fbb | Initial commit | ✅ PASS | | 1f7e8c5 | add default config | ✅ PASS | | **63e0c22** | **change port** | ❌ **FAIL** ← first failure | | cf3fea9 | add feature flag | ❌ FAIL | | 3191943 | add port constant | ❌ FAIL | The build transitions from **PASS → FAIL** exactly at commit `63e0c22`. Every commit before it passes, and every commit from it onward fails. This is the classic bisection signature of a bug-introducing commit — `63e0c22` ("change port") is the culprit.
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 522 in → 15 out (0 reasoning)
61e18d4
JSON Schema Structured Output
(raw) {"commit": "61e18d4"}
anthropic/claude-sonnet-4.6:high PASS 100% 224 in → 29 out (0 reasoning)
The bad commit is **71a906d** ("change port"), as it's the first commit where the tests began failing.
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 406 in → 12 out (0 reasoning)
eae38f0
JSON Schema Structured Output
(raw) {"commit":"eae38f0"}
anthropic/claude-sonnet-4.6:low PASS 100% 232 in → 7 out (0 reasoning)
95608af
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 408 in → 12 out (0 reasoning)
7d69b78
JSON Schema Structured Output
(raw) {"commit":"7d69b78"}
anthropic/claude-sonnet-4.6:max PASS 100% 238 in → 385 out (127 reasoning)
## Analysis Looking at the test results ordered from oldest to newest: | Commit | Subject | Result | |--------|---------|--------| | `4f6a54e` | Initial commit | ✅ PASS | | `d7df229` | add default config | ✅ PASS | | `53ebe29` | change port | ❌ FAIL | | `5b714a2` | add feature flag | ❌ FAIL | | `ef46624` | add port constant | ❌ FAIL | The build passes through `d7df229` and first breaks at `53ebe29`. All subsequent commits also fail, which is consistent with a bug being introduced and never fixed. ## Answer **`53ebe29`** The bad commit is `53ebe29` — **"change port"** — which is the first commit where the build transitions from PASS to FAIL.
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 400 in → 96 out (57 reasoning)
c141677
JSON Schema Structured Output
(raw) {"commit":"c141677"}
anthropic/claude-sonnet-4.6:medium PASS 100% 240 in → 13 out (0 reasoning)
The bad commit is **83bce98**
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 408 in → 11 out (0 reasoning)
50d049f
JSON Schema Structured Output
(raw) {"commit":"50d049f"}
anthropic/claude-sonnet-4.6:none PASS 100% 246 in → 14 out (0 reasoning)
The bad commit is **0f6ad73**
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 406 in → 10 out (0 reasoning)
4213655
JSON Schema Structured Output
(raw) {"commit":"4213655"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 240 in → 30 out (0 reasoning)
The bad commit is **f8984d1** ("change port"), as it is the first commit where the tests began failing.
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 402 in → 11 out (0 reasoning)
6cd5262
JSON Schema Structured Output
(raw) {"commit":"6cd5262"}
anthropic/claude-sonnet-5:high PASS 100% 305 in → 82 out (0 reasoning)
**4964d3e** — "change port" This is the first commit where the test transitioned from PASS to FAIL, and all subsequent commits remain FAIL, indicating this commit introduced the breaking change (and later commits simply inherited the broken state).
anthropic/claude-sonnet-5:high__json_schema PASS 100% 526 in → 14 out (0 reasoning)
da26a40
JSON Schema Structured Output
(raw) {"commit":"da26a40"}
anthropic/claude-sonnet-5:low PASS 100% 297 in → 10 out (0 reasoning)
6ab21fd change port
anthropic/claude-sonnet-5:low__json_schema PASS 100% 520 in → 15 out (0 reasoning)
1e06d02
JSON Schema Structured Output
(raw) {"commit":"1e06d02"}
anthropic/claude-sonnet-5:medium PASS 100% 301 in → 8 out (0 reasoning)
c1e10b3
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 514 in → 15 out (0 reasoning)
6fe2893
JSON Schema Structured Output
(raw) {"commit":"6fe2893"}
anthropic/claude-sonnet-5:none PASS 100% 297 in → 9 out (0 reasoning)
c247c18 change port
anthropic/claude-sonnet-5:none__json_schema PASS 100% 516 in → 13 out (0 reasoning)
58605ee
JSON Schema Structured Output
(raw) {"commit":"58605ee"}
anthropic/claude-sonnet-5:xhigh PASS 100% 299 in → 127 out (0 reasoning)
**fa13c82** (commit subject: "change port") This is the first commit where the test suite transitions from PASS to FAIL, and all subsequent commits remain in a FAIL state — consistent with a bug introduced at this point that was never fixed. Using a bisection approach: since `67cc397` passes and `fa13c82` fails, the bug was introduced in `fa13c82`.
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 514 in → 15 out (0 reasoning)
597ad8d
JSON Schema Structured Output
(raw) {"commit":"597ad8d"}
arcee-ai/trinity-large-thinking:high PASS 100% 219 in → 1,057 out (1,060 reasoning)
88e01a5
arcee-ai/trinity-large-thinking:low PASS 100% 217 in → 869 out (843 reasoning)
The first commit that caused a test failure is `ec54197` with the subject line "change port". This is the bad commit that introduced the bug.
arcee-ai/trinity-large-thinking:medium PASS 100% 221 in → 976 out (924 reasoning)
The first commit that causes the test to fail is `4206cd3` ("change port"). Since the build was passing in previous commits and all subsequent commits also fail, this is the bad commit that introduced the bug. Answer: 4206cd3
arcee-ai/trinity-large-thinking:xhigh PASS 100% 217 in → 1,118 out (1,073 reasoning)
The bad commit is `a8c0038` with the subject line "change port", as it is the first commit where the tests fail (exit code 1), indicating the introduction of the bug.
arcee-ai/trinity-mini:high PASS 100% 215 in → 767 out (489 reasoning)
24c9788**Reasoning:** 1. **Test Results Order:** The test results are listed from oldest commit to newest. This means the first test result (e0808de) is the oldest commit, and the last test result (18b1429) is the newest. 2. **Failure Pattern:** The test results show that commits `e0808de` and `8035a76` passed (exit code 0). All commits after that (`24c9788`, `3566a0b`, `18b1429`) failed (exit code 1). 3. **First Failure:** The first commit to fail is `24c9788` ("change port"). Since the build was working before this commit (as `e0808de` and `8035a76` passed), this commit is the one that introduced the bug. 4. **Subsequent Failures:** While commits `3566a0b` ("add feature flag") and `18b1429` ("add port constant") also failed, the bug was already introduced in `24c9788`. Their failures are a consequence of the existing bug, not the initial cause. Therefore, the bad commit that broke the build is `24c9788`.
arcee-ai/trinity-mini:high__json_schema PASS 100% 213 in → 225 out (216 reasoning)
c61b131
JSON Schema Structured Output
(raw) { "commit": "c61b131" }
arcee-ai/trinity-mini:low PASS 100% 217 in → 663 out (636 reasoning)
3c2127c
arcee-ai/trinity-mini:low__json_schema PASS 100% 219 in → 447 out (407 reasoning)
6f6666d
JSON Schema Structured Output
(raw) { "commit": "6f6666d" }
arcee-ai/trinity-mini:medium PASS 100% 217 in → 523 out (343 reasoning)
033e4de **Reasoning:** 1. **Test Results Order:** The test results are listed from oldest to newest commit. 2. **Pass/Fail Pattern:** The first two commits (9b35f91 and e3ca556) passed. The third commit (033e4de) failed. The subsequent commits (3ec7f16 and ae56a64) also failed. 3. **Root Cause:** The failure starts at commit `033e4de`. This is the first commit where the test fails. While commits `3ec7f16` and `ae56a64` also fail, they are likely failing *because* the code was already broken by `033e4de`. The bug was introduced in `033e4de`. 4. **Bad Commit:** Therefore, the commit that introduced the bug is `033e4de`.
arcee-ai/trinity-mini:medium__json_schema PASS 100% 221 in → 670 out (598 reasoning)
4ac9e9f
JSON Schema Structured Output
(raw) { "commit": "4ac9e9f" }
arcee-ai/trinity-mini:xhigh PASS 100% 219 in → 411 out (370 reasoning)
2b86c68
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 213 in → 900 out (909 reasoning)
03d8836
JSON Schema Structured Output
(raw) { "commit": "03d8836" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 289 in → 62 out (53 reasoning)
4146448
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 301 in → 94 out (70 reasoning)
9be8203
JSON Schema Structured Output
(raw) { "commit": "9be8203" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 220 in → 274 out (267 reasoning)
f4795f4
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 206 in → 279 out (267 reasoning)
931fb35
JSON Schema Structured Output
(raw) {"commit":"931fb35"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 310 in → 306 out (300 reasoning)
9451f6e
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 291 in → 189 out (165 reasoning)
f5473ab
JSON Schema Structured Output
(raw) {"commit": "f5473ab"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 216 in → 256 out (216 reasoning)
10280fa
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 208 in → 159 out (174 reasoning)
eb86408
JSON Schema Structured Output
(raw) { "commit": "eb86408" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 218 in → 6 out (0 reasoning)
7890a7b
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 208 in → 9 out (0 reasoning)
88882e2
JSON Schema Structured Output
(raw) {"commit":"88882e2"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 208 in → 159 out (146 reasoning)
d3cef09
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 285 in → 149 out (126 reasoning)
a707a10
JSON Schema Structured Output
(raw) { "commit": "a707a10" }
deepseek/deepseek-v4-flash:high PASS 100% 222 in → 199 out (191 reasoning)
1e6ef2f
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 310 in → 10 out (0 reasoning)
4b5530e
JSON Schema Structured Output
(raw) {"commit":"4b5530e"}
deepseek/deepseek-v4-flash:low PASS 100% 204 in → 78 out (72 reasoning)
298f09c
deepseek/deepseek-v4-flash:medium PASS 100% 202 in → 104 out (99 reasoning)
6146405
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 282 in → 78 out (54 reasoning)
35c0b42
JSON Schema Structured Output
(raw) {"commit": "35c0b42"}
deepseek/deepseek-v4-flash:none PASS 100% 210 in → 62 out (0 reasoning)
Based on the git log and test results, the first commit that fails is `2ccac79` "change port". Since commits before it pass (0444e51 and 2b83c53), and this commit introduces the failure, it is the bad commit. **2ccac79**
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 208 in → 10 out (0 reasoning)
7387541
JSON Schema Structured Output
(raw) {"commit":"7387541"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 297 in → 286 out (279 reasoning)
de2d08a
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 295 in → 125 out (112 reasoning)
2020dd9
JSON Schema Structured Output
(raw) { "commit": "2020dd9" }
deepseek/deepseek-v4-pro:high PASS 100% 218 in → 193 out (182 reasoning)
a25153f
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 397 in → 103 out (93 reasoning)
39452bc
JSON Schema Structured Output
(raw) {"commit": "39452bc"}
deepseek/deepseek-v4-pro:low PASS 100% 216 in → 144 out (137 reasoning)
d7a0d89
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 407 in → 77 out (65 reasoning)
cb1e53b
JSON Schema Structured Output
(raw) {"commit": "cb1e53b"}
deepseek/deepseek-v4-pro:medium PASS 100% 218 in → 168 out (153 reasoning)
b85dd0e
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 409 in → 88 out (77 reasoning)
1441cc6
JSON Schema Structured Output
(raw) {"commit": "1441cc6"}
deepseek/deepseek-v4-pro:none PASS 100% 222 in → 46 out (0 reasoning)
Based on the test results, the bad commit is: **a945a49** This is the first commit where the tests failed, indicating it introduced the bug. All subsequent commits also fail because the bug persisted through those changes.
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 220 in → 9 out (0 reasoning)
dacca9d
JSON Schema Structured Output
(raw) {"commit": "dacca9d"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 295 in → 337 out (332 reasoning)
9093a34
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 293 in → 195 out (180 reasoning)
5452f3e
JSON Schema Structured Output
(raw) { "commit": "5452f3e" }
google/gemini-3-flash-preview:high PASS 100% 237 in → 186 out (172 reasoning)
The bad commit is 6e9cbc5 or change port.
google/gemini-3-flash-preview:high__json_schema PASS 100% 240 in → 321 out (310 reasoning)
1b2bb37
JSON Schema Structured Output
(raw) {"commit": "1b2bb37"}
google/gemini-3-flash-preview:low PASS 100% 235 in → 271 out (256 reasoning)
The commit that introduced the bug is 55b1bf7.
google/gemini-3-flash-preview:low__json_schema PASS 100% 304 in → 208 out (198 reasoning)
89f5bd3
JSON Schema Structured Output
(raw) {"commit":"89f5bd3"}
google/gemini-3-flash-preview:medium PASS 100% 239 in → 183 out (171 reasoning)
The bad commit is 2a3da60.
google/gemini-3-flash-preview:medium__json_schema PASS 100% 298 in → 264 out (253 reasoning)
04f5208
JSON Schema Structured Output
(raw) {"commit":"04f5208"}
google/gemini-3-flash-preview:xhigh PASS 100% 231 in → 224 out (219 reasoning)
3eddd05
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 300 in → 508 out (498 reasoning)
5a8ff77
JSON Schema Structured Output
(raw) {"commit":"5a8ff77"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 241 in → 595 out (512 reasoning)
The bad commit is **2a8b793** (subject: **change port**). **Reasoning:** The test results show that `b97a540` was the last commit to pass (PASS). The very next commit, `2a8b793`, was the first to fail (FAIL), indicating that the bug was introduced in that specific commit.
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 246 in → 298 out (286 reasoning)
2e1473e
JSON Schema Structured Output
(raw) {"commit": "2e1473e"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 236 in → 257 out (154 reasoning)
The bad commit is **fa3f876** (subject line: "change port"). **Reasoning:** Looking at the test results chronologically: * `ef1ea4e` (the commit before `fa3f876`) passed. * `fa3f876` failed. Since the build was passing before this commit and failed started at this commit, `fa3f876` is the commit that introduced the bug.
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 296 in → 134 out (123 reasoning)
dff8f40
JSON Schema Structured Output
(raw) {"commit": "dff8f40"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 240 in → 257 out (240 reasoning)
The bad commit is **35b3c78** (change port).
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 292 in → 228 out (212 reasoning)
96214aa
JSON Schema Structured Output
(raw) { "commit": "96214aa" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 234 in → 581 out (503 reasoning)
The bad commit is **75c78dc** (subject: "change port"). **Reasoning:** The test results show that commit `832f0c6` passed, but the very next commit, `75c78dc`, failed. This indicates that the change introduced in `75c78dc` caused the build failure.
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 302 in → 239 out (227 reasoning)
5c42104
JSON Schema Structured Output
(raw) {"commit": "5c42104"}
google/gemini-3.1-pro-preview:high PASS 100% 235 in → 375 out (216 reasoning)
To find the commit that introduced the bug, we can look at the test results in sequential order, from the oldest commit to the newest. The first commit to fail the test is the one that broke the build. Looking at your test results: - 2c7b328 Initial commit: PASS - 2d8ebce add default config: PASS - **ad98c36 change port: FAIL** - e05c3b9 add feature flag: FAIL - eab2aa7 add port constant: FAIL The build was passing until the "change port" commit. Therefore, the commit that introduced the bug is: **ad98c36** (or "change port")
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 300 in → 154 out (143 reasoning)
dfb9479
JSON Schema Structured Output
(raw) {"commit": "dfb9479"}
google/gemini-3.1-pro-preview:low PASS 100% 241 in → 174 out (152 reasoning)
The bad commit that introduced the bug is **da5d081** (or "change port").
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 300 in → 69 out (53 reasoning)
431fb64
JSON Schema Structured Output
(raw) { "commit": "431fb64" }
google/gemini-3.1-pro-preview:medium PASS 100% 239 in → 192 out (172 reasoning)
The bad commit that introduced the bug is **43b0bb4** ("change port").
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 300 in → 173 out (157 reasoning)
5ed6e66
JSON Schema Structured Output
(raw) { "commit": "5ed6e66" }
google/gemini-3.1-pro-preview:xhigh PASS 100% 241 in → 277 out (188 reasoning)
The bad commit that introduced the bug is **459cde1** (or "change port"). Looking at the chronological test results (oldest first), the build passes for the first two commits (`2c77517` and `b651705`) but fails immediately when `459cde1 change port` is introduced. All subsequent commits also fail over this broken state.
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 236 in → 290 out (280 reasoning)
ec1828c
JSON Schema Structured Output
(raw) {"commit":"ec1828c"}
google/gemini-3.5-flash-lite:high PASS 100% 241 in → 476 out (470 reasoning)
df3a88e
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 296 in → 516 out (506 reasoning)
ed4bc60
JSON Schema Structured Output
(raw) {"commit": "ed4bc60"}
google/gemini-3.5-flash-lite:low PASS 100% 242 in → 122 out (115 reasoning)
c674561
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 294 in → 16 out (0 reasoning)
a3bd4d5
JSON Schema Structured Output
(raw) { "commit": "a3bd4d5" }
google/gemini-3.5-flash-lite:max PASS 100% 239 in → 487 out (481 reasoning)
be03a88
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 300 in → 444 out (435 reasoning)
81acfa7
JSON Schema Structured Output
(raw) {"commit":"81acfa7"}
google/gemini-3.5-flash-lite:medium PASS 100% 242 in → 195 out (189 reasoning)
6ab76e0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 238 in → 301 out (289 reasoning)
f64085d
JSON Schema Structured Output
(raw) {"commit": "f64085d"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 238 in → 443 out (437 reasoning)
c97ebf9
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 302 in → 351 out (340 reasoning)
619898c
JSON Schema Structured Output
(raw) {"commit":"619898c"}
google/gemini-3.5-flash:high PASS 100% 237 in → 229 out (222 reasoning)
c346479
google/gemini-3.5-flash:high__json_schema PASS 100% 292 in → 338 out (328 reasoning)
50d76ed
JSON Schema Structured Output
(raw) {"commit":"50d76ed"}
google/gemini-3.5-flash:low PASS 100% 239 in → 132 out (127 reasoning)
9acbf2b
google/gemini-3.5-flash:low__json_schema PASS 100% 294 in → 117 out (106 reasoning)
0ce2221
JSON Schema Structured Output
(raw) {"commit": "0ce2221"}
google/gemini-3.5-flash:medium PASS 100% 231 in → 260 out (254 reasoning)
af899d4
google/gemini-3.5-flash:medium__json_schema PASS 100% 304 in → 298 out (288 reasoning)
bb5e826
JSON Schema Structured Output
(raw) {"commit":"bb5e826"}
google/gemini-3.5-flash:xhigh PASS 100% 241 in → 310 out (304 reasoning)
65949ee
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 304 in → 411 out (399 reasoning)
a7d237e
JSON Schema Structured Output
(raw) {"commit": "a7d237e"}
google/gemini-3.6-flash:high PASS 100% 241 in → 329 out (306 reasoning)
The bad commit that introduced the bug is **`8a047e0`** (change port).
google/gemini-3.6-flash:high__json_schema PASS 100% 304 in → 345 out (335 reasoning)
bcae38a
JSON Schema Structured Output
(raw) {"commit":"bcae38a"}
google/gemini-3.6-flash:low PASS 100% 242 in → 109 out (88 reasoning)
The bad commit that introduced the bug is **91800c7** (change port).
google/gemini-3.6-flash:low__json_schema PASS 100% 240 in → 103 out (92 reasoning)
9a03b68
JSON Schema Structured Output
(raw) {"commit":"9a03b68"}
google/gemini-3.6-flash:max PASS 100% 241 in → 284 out (260 reasoning)
**87f000e** (or "change port") is the bad commit that introduced the bug.
google/gemini-3.6-flash:max__json_schema PASS 100% 292 in → 274 out (264 reasoning)
04c91fe
JSON Schema Structured Output
(raw) {"commit":"04c91fe"}
google/gemini-3.6-flash:medium PASS 100% 246 in → 291 out (271 reasoning)
The commit that introduced the bug is **d1010e1** (`change port`).
google/gemini-3.6-flash:medium__json_schema PASS 100% 306 in → 193 out (182 reasoning)
18d2b2d
JSON Schema Structured Output
(raw) {"commit":"18d2b2d"}
google/gemini-3.6-flash:xhigh PASS 100% 235 in → 222 out (200 reasoning)
The commit that introduced the bug is **384a0d8** (or "change port").
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 242 in → 410 out (399 reasoning)
258038b
JSON Schema Structured Output
(raw) {"commit":"258038b"}
google/gemma-4-26b-a4b-it:high PASS 100% 253 in → 490 out (337 reasoning)
022dee0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 251 in → 493 out (317 reasoning)
9ada336
JSON Schema Structured Output
(raw) { "commit": "9ada336" }
google/gemma-4-26b-a4b-it:low PASS 100% 253 in → 325 out (279 reasoning)
61c74f2
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 248 in → 253 out (148 reasoning)
17dd6ef
JSON Schema Structured Output
(raw) {"commit": "17dd6ef"}
google/gemma-4-26b-a4b-it:medium PASS 100% 250 in → 251 out (166 reasoning)
58e75b1
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 251 in → 347 out (233 reasoning)
604b7ba
JSON Schema Structured Output
(raw) { "commit": "604b7ba" }
google/gemma-4-26b-a4b-it:none PASS 100% 252 in → 8 out (0 reasoning)
5802375
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 246 in → 16 out (0 reasoning)
afe4915
JSON Schema Structured Output
(raw) { "commit": "afe4915" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 248 in → 364 out (238 reasoning)
f2a8a94
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 255 in → 309 out (186 reasoning)
87616d1
JSON Schema Structured Output
(raw) { "commit": "87616d1" }
google/gemma-4-31b-it:high PASS 100% 253 in → 186 out (116 reasoning)
a8b037c
google/gemma-4-31b-it:high__json_schema PASS 100% 259 in → 164 out (80 reasoning)
e3b1c58
JSON Schema Structured Output
(raw) {"commit": "e3b1c58"}
google/gemma-4-31b-it:low PASS 100% 253 in → 150 out (103 reasoning)
55e9dcf
google/gemma-4-31b-it:low__json_schema PASS 100% 258 in → 163 out (101 reasoning)
3a85262
JSON Schema Structured Output
(raw) { "commit": "3a85262" }
google/gemma-4-31b-it:medium PASS 100% 253 in → 233 out (168 reasoning)
ffd8899
google/gemma-4-31b-it:medium__json_schema PASS 100% 244 in → 220 out (142 reasoning)
89cb450
JSON Schema Structured Output
(raw) {"commit": "89cb450"}
google/gemma-4-31b-it:none PASS 100% 252 in → 7 out (0 reasoning)
c2db501
google/gemma-4-31b-it:none__json_schema PASS 100% 250 in → 10 out (0 reasoning)
bc1cafb
JSON Schema Structured Output
(raw) {"commit": "bc1cafb"}
google/gemma-4-31b-it:xhigh PASS 100% 255 in → 361 out (1 reasoning)
48b400b
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 248 in → 183 out (106 reasoning)
cbbf970
JSON Schema Structured Output
(raw) {"commit": "cbbf970"}
ibm-granite/granite-4.1-8b PASS 100% 226 in → 76 out
The failing test results indicate that the build started to fail with commit **fd549ee** ("change port"). While subsequent commits (8a9ca30 and eb4f8b8) also cause failures, the first occurrence of a failure marks the introduction of the bug. Therefore, the bad commit hash (first 7 characters) is: **fd549ee**
ibm-granite/granite-4.1-8b__json_schema PASS 100% 234 in → 11 out
b65edc6
JSON Schema Structured Output
(raw) {"commit": "b65edc6"}
inclusionai/ling-2.6-flash PASS 100% 257 in → 65 out
The bad commit is **0548f17** ("change port"). The test results show that the first failure occurs at commit `0548f17` ("change port"). All earlier commits pass, and all later commits also fail — indicating the bug was introduced by this commit.
inclusionai/ling-2.6-flash__json_schema PASS 100% 257 in → 23 out
b3c7e27 (b3c7e27 change port)
JSON Schema Structured Output
(raw) {"commit":"b3c7e27 (b3c7e27 change port)"}
liquid/lfm-2-24b-a2b PASS 100% 237 in → 38 out
The first failing commit after the initial pass is **35bde5a**, as it is the earliest commit that produced a failure. Answer: 35bde5a
minimax/minimax-m2.5:high PASS 100% 244 in → 418 out (418 reasoning)
a823384
minimax/minimax-m2.5:high__json_schema PASS 100% 238 in → 383 out (351 reasoning)
1f5264f
JSON Schema Structured Output
(raw) { "commit": "1f5264f" }
minimax/minimax-m2.5:low PASS 100% 254 in → 212 out (205 reasoning)
da53b2c
minimax/minimax-m2.5:low__json_schema PASS 100% 252 in → 330 out (293 reasoning)
b5c08c4
JSON Schema Structured Output
(raw) { "commit": "b5c08c4" }
minimax/minimax-m2.5:medium PASS 100% 248 in → 278 out (266 reasoning)
f5fcc1c (commit “change port”)
minimax/minimax-m2.5:medium__json_schema PASS 100% 244 in → 333 out (316 reasoning)
907ecbc
JSON Schema Structured Output
(raw) { "commit": "907ecbc" }
minimax/minimax-m2.5:xhigh PASS 100% 240 in → 236 out (230 reasoning)
9ea488f
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 259 in → 169 out (157 reasoning)
ff960ab
JSON Schema Structured Output
(raw) { "commit": "ff960ab" }
minimax/minimax-m2.7:high PASS 100% 242 in → 416 out (410 reasoning)
911855e
minimax/minimax-m2.7:high__json_schema PASS 100% 236 in → 13 out (13 reasoning)
57cd797
JSON Schema Structured Output
(raw) { "commit": "57cd797" }
minimax/minimax-m2.7:low PASS 100% 247 in → 168 out (163 reasoning)
05c4487
minimax/minimax-m2.7:low__json_schema PASS 100% 385 in → 317 out (306 reasoning)
9bdf49e
JSON Schema Structured Output
(raw) {"commit":"9bdf49e"}
minimax/minimax-m2.7:medium PASS 100% 245 in → 271 out (247 reasoning)
4fa8226 (the “change port” commit) is the first one that causes the test to fail.
minimax/minimax-m2.7:medium__json_schema PASS 100% 385 in → 323 out (310 reasoning)
9e0f5d5
JSON Schema Structured Output
(raw) {"commit":"9e0f5d5"}
minimax/minimax-m2.7:xhigh PASS 100% 247 in → 434 out (428 reasoning)
14d84b7
minimax/minimax-m3:high PASS 100% 378 in → 116 out (96 reasoning)
The bad commit is `2aca6bd` (change port).
minimax/minimax-m3:high__json_schema PASS 100% 377 in → 12 out (0 reasoning)
26e2a6d
JSON Schema Structured Output
(raw) {"commit": "26e2a6d"}
minimax/minimax-m3:low PASS 100% 384 in → 88 out (66 reasoning)
cce0ac2
minimax/minimax-m3:low__json_schema PASS 100% 367 in → 10 out (0 reasoning)
5ad5efe
JSON Schema Structured Output
(raw) {"commit": "5ad5efe"}
minimax/minimax-m3:medium PASS 100% 388 in → 121 out (113 reasoning)
a41bf2e
minimax/minimax-m3:medium__json_schema PASS 100% 367 in → 11 out (0 reasoning)
51099ec
JSON Schema Structured Output
(raw) { "commit": "51099ec" }
minimax/minimax-m3:xhigh PASS 100% 384 in → 128 out (109 reasoning)
6fa3493
minimax/minimax-m3:xhigh__json_schema PASS 100% 369 in → 12 out (0 reasoning)
763517f
JSON Schema Structured Output
(raw) { "commit": "763517f" }
mistralai/devstral-2512 PASS 100% 235 in → 92 out
Based on the test results, the first commit that introduced the bug is `568c638 change port`, as it's the first commit in the history (when read from oldest to newest) that shows a FAIL result. All subsequent commits also fail, which suggests the bug was introduced in this commit and not fixed in later ones. The bad commit is: `568c638` (or "change port")
mistralai/devstral-2512__json_schema PASS 100% 245 in → 13 out
64587a0
JSON Schema Structured Output
(raw) {"commit": "64587a0"}
mistralai/mistral-medium-3-5:high PASS 100% 251 in → 512 out (334 reasoning)
The test results show that the build was passing up to commit `68c4469` (add default config) and first failed at commit `d86431b` (change port). All subsequent commits also failed, confirming that `d86431b` is the first bad commit. **d86431b**
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 237 in → 288 out (215 reasoning)
08a7efb
JSON Schema Structured Output
(raw) {"commit": "08a7efb"}
mistralai/mistral-medium-3-5:low PASS 100% 255 in → 529 out (389 reasoning)
b224b30
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 239 in → 269 out (206 reasoning)
c79d362
JSON Schema Structured Output
(raw) {"commit": "c79d362"}
mistralai/mistral-medium-3-5:medium PASS 100% 251 in → 405 out (272 reasoning)
The first failing commit is `c74dee0 change port`, and all subsequent commits also fail. Since the two commits before it (`7c42a6e` and `06d7f35`) pass, **`c74dee0` is the bad commit that broke the build**. c74dee0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 237 in → 370 out (264 reasoning)
bb44116
JSON Schema Structured Output
(raw) {"commit": "bb44116"}
mistralai/mistral-medium-3-5:none PASS 100% 251 in → 122 out (0 reasoning)
The bad commit is `866d255` (commit subject: "change port"). This is because: 1. The test passes for commits 92b49cc and ec4870e 2. The test first fails at commit 866d255 3. All subsequent commits (459760d and f0bd616) also fail, indicating the bug was introduced in 866d255 and never fixed Therefore, the bad commit is `866d255`.
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 237 in → 13 out (0 reasoning)
133805c
JSON Schema Structured Output
(raw) {"commit": "133805c"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 253 in → 490 out (338 reasoning)
The test passes for the first two commits (Initial commit and add default config) but starts failing at **86a96d1 change port** and continues to fail for all subsequent commits. This indicates the bug was introduced in that commit. 86a96d1
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 231 in → 370 out (311 reasoning)
af9feed
JSON Schema Structured Output
(raw) {"commit": "af9feed"}
moonshotai/kimi-k2.6:high PASS 100% 212 in → 254 out (248 reasoning)
3e12858
moonshotai/kimi-k2.6:high__json_schema PASS 100% 212 in → 345 out (334 reasoning)
ccdaa6c
JSON Schema Structured Output
(raw) { "commit": "ccdaa6c" }
moonshotai/kimi-k2.6:low PASS 100% 208 in → 432 out (427 reasoning)
bdd7864
moonshotai/kimi-k2.6:low__json_schema PASS 100% 219 in → 808 out (281 reasoning)
68fe6b8
JSON Schema Structured Output
(raw) { "commit": "68fe6b8" }
moonshotai/kimi-k2.6:medium PASS 100% 216 in → 387 out (384 reasoning)
180012a
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 215 in → 568 out (343 reasoning)
e92410b
JSON Schema Structured Output
(raw) { "commit": "e92410b" }
moonshotai/kimi-k2.6:xhigh PASS 100% 228 in → 336 out (328 reasoning)
7c5e85a
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 215 in → 725 out (374 reasoning)
6f1bd4c
JSON Schema Structured Output
(raw) { "commit": "6f1bd4c" }
moonshotai/kimi-k2.7-code:high PASS 100% 210 in → 133 out (124 reasoning)
587431d (change port)
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 216 in → 231 out (215 reasoning)
b2fabe0 change port
JSON Schema Structured Output
(raw) { "commit": "b2fabe0 change port" }
moonshotai/kimi-k2.7-code:low PASS 100% 214 in → 92 out (86 reasoning)
1db698f
moonshotai/kimi-k2.7-code:medium PASS 100% 218 in → 122 out (112 reasoning)
34293f4 (change port)
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 214 in → 160 out (144 reasoning)
aab009a change port
JSON Schema Structured Output
(raw) { "commit": "aab009a change port" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 213 in → 163 out (131 reasoning)
**64a412c** — “change port” This is the first commit in the test results that fails, so it introduced the bug.
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 216 in → 183 out (171 reasoning)
6deddd5
JSON Schema Structured Output
(raw) { "commit": "6deddd5" }
moonshotai/kimi-k3:high PASS 100% 293 in → 257 out (87 reasoning)
**Bad commit: `7d91952` — "change port"** **Reasoning:** - `8418c12` (Initial commit) → PASS - `fec8d96` (add default config) → PASS - `7d91952` (change port) → **FAIL** ← first failure appears here - `aa9a67b` (add feature flag) → FAIL - `2e250f3` (add port constant) → FAIL Since the build passed at `fec8d96` and first failed at `7d91952`, the commit `7d91952` ("change port") is the one that introduced the bug. All subsequent commits fail because they build on top of it.
moonshotai/kimi-k3:high__json_schema PASS 100% 295 in → 223 out (197 reasoning)
3fd5f9d — “change port”
moonshotai/kimi-k3:low PASS 100% 295 in → 251 out (95 reasoning)
**Bad commit: `3e5ff9c` — "change port"** **Reasoning:** The build passed at `0be0239` (Initial commit) and `9059f8d` (add default config), but began failing at `3e5ff9c` (change port). Every commit after it also fails, which is consistent with the bug being introduced at `3e5ff9c` and persisting through all subsequent commits. Since `9059f8d` is the last passing commit and `3e5ff9c` is the first failing one, `3e5ff9c` is the commit that introduced the bug.
moonshotai/kimi-k3:low__json_schema PASS 100% 297 in → 70 out (44 reasoning)
47b7e6a — “change port”
moonshotai/kimi-k3:max PASS 100% 295 in → 268 out (67 reasoning)
**Bad commit: `6c2bdb2` — "change port"** **Reasoning:** Walking the history from oldest to newest: - `ac52625` (Initial commit) — PASS - `cea3e3f` (add default config) — PASS - `6c2bdb2` (change port) — **FAIL** ← first failure - `0d38f59` and `8b2dcd5` — FAIL (they inherit the breakage) Since `cea3e3f` is the last passing commit and `6c2bdb2` is the first failing one, the bug was introduced in `6c2bdb2` ("change port"). The subsequent commits remain broken because they build on top of this bad change — notably, the later commit "add port constant" (`8b2dcd5`) didn't fix the port change that broke things.
moonshotai/kimi-k3:max__json_schema PASS 100% 289 in → 195 out (75 reasoning)
**Bad commit: `9785417` — "change port"** The tests pass on the first two commits (`099c6ca` and `f0ac59d`), but `9785417` is the first commit where the test suite fails (exit code 1). Since every subsequent commit also fails, `9785417` is the commit that introduced the bug. In a `git bisect` workflow, this is exactly what bisection would converge on: the transition point from PASS to FAIL.
moonshotai/kimi-k3:medium PASS 100% 295 in → 305 out (91 reasoning)
**Bad commit: `f3f268d` ("change port")** **Reasoning:** This follows the standard git bisect logic — the bad commit is the *first* commit where the test transitions from PASS to FAIL: | Commit | Result | |--------|--------| | `5135b3c` Initial commit | ✅ PASS | | `3f0417f` add default config | ✅ PASS | | **`f3f268d` change port** | ❌ **FAIL** ← first failure | | `02a8090` add feature flag | ❌ FAIL | | `4a477bd` add port constant | ❌ FAIL | Since `3f0417f` passes and `f3f268d` is the first commit to fail — with all subsequent commits also failing — the bug was introduced in **`f3f268d` ("change port")**. The later commits didn't break the build; they just inherited the breakage.
moonshotai/kimi-k3:medium__json_schema PASS 100% 293 in → 295 out (116 reasoning)
**Bad commit: `c80ca9b` ("change port")** **Reasoning:** - `7872e04` (Initial commit) → PASS - `8b1fa3e` (add default config) → PASS - `c80ca9b` (change port) → **FAIL** ← first failure - `52f414c` (add feature flag) → FAIL - `5866934` (add port constant) → FAIL The tests pass through `8b1fa3e` and first fail at `c80ca9b`. Since the bug appears in every commit from `c80ca9b` onward and in none before it, `c80ca9b` ("change port") is the commit that introduced the bug.
moonshotai/kimi-k3:xhigh PASS 100% 297 in → 115 out (91 reasoning)
7580a34 — “change port”
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 293 in → 325 out (136 reasoning)
**Bad commit: `67d36d9` — "change port"** **Reasoning:** Walking the history from oldest to newest: - `292f6dc` (Initial commit): PASS - `89c2ea8` (add default config): PASS - `67d36d9` (change port): **FAIL** ← first failure - `007502d` (add feature flag): FAIL - `a3db15d` (add port constant): FAIL Since `89c2ea8` is the last passing commit and `67d36d9` is the first failing commit — with every subsequent commit also failing — the bug was introduced in `67d36d9` ("change port"). This matches a classic git bisect result: the transition point from PASS to FAIL identifies the culprit.
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 254 in → 292 out (251 reasoning)
2ba2f4d
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 248 in → 181 out (156 reasoning)
e2ad374
JSON Schema Structured Output
(raw) { "commit": "e2ad374" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 250 in → 360 out (360 reasoning)
1824dc4
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 250 in → 380 out (288 reasoning)
85a88cc
JSON Schema Structured Output
(raw) {"commit":"85a88cc"}
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 248 in → 471 out (477 reasoning)
**f9f8ffc** (the “change port” commit)
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 258 in → 374 out (357 reasoning)
15f0b0d
JSON Schema Structured Output
(raw) { "commit": "15f0b0d" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 250 in → 9 out (0 reasoning)
2b5bc53 change port
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 248 in → 14 out (0 reasoning)
af85a59
JSON Schema Structured Output
(raw) { "commit": "af85a59" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 252 in → 156 out (135 reasoning)
03097d6 (change port)
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 252 in → 377 out (295 reasoning)
dc1c0ee
JSON Schema Structured Output
(raw) { "commit": "dc1c0ee" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 246 in → 115 out (78 reasoning)
3036ff3
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 250 in → 216 out (184 reasoning)
0f2dc4a
JSON Schema Structured Output
(raw) { "commit": "0f2dc4a" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 252 in → 173 out (138 reasoning)
2c60686 change port
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 256 in → 238 out (168 reasoning)
c812775
JSON Schema Structured Output
(raw) { "commit": "c812775" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 254 in → 162 out (142 reasoning)
744d8d3
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 248 in → 98 out (81 reasoning)
e1c5c3d
JSON Schema Structured Output
(raw) { "commit": "e1c5c3d" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 244 in → 8 out (0 reasoning)
b5dceab change port
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 258 in → 16 out (0 reasoning)
1575838
JSON Schema Structured Output
(raw) { "commit": "1575838" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 252 in → 180 out (156 reasoning)
1952ef8
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 258 in → 251 out (203 reasoning)
b4e3665
JSON Schema Structured Output
(raw) { "commit": "b4e3665" }
openai/gpt-5.4-mini:high PASS 100% 208 in → 71 out (61 reasoning)
875b92d
openai/gpt-5.4-mini:high__json_schema PASS 100% 251 in → 110 out (92 reasoning)
33743bb
JSON Schema Structured Output
(raw) {"commit":"33743bb"}
openai/gpt-5.4-mini:low PASS 100% 218 in → 51 out (39 reasoning)
9c0c2ea
openai/gpt-5.4-mini:low__json_schema PASS 100% 253 in → 53 out (33 reasoning)
f1d8845
JSON Schema Structured Output
(raw) {"commit":"f1d8845"}
openai/gpt-5.4-mini:medium PASS 100% 204 in → 49 out (38 reasoning)
7b9ca16
openai/gpt-5.4-mini:medium__json_schema PASS 100% 257 in → 53 out (33 reasoning)
16c1ca8
JSON Schema Structured Output
(raw) {"commit":"16c1ca8"}
openai/gpt-5.4-mini:none PASS 100% 216 in → 8 out (0 reasoning)
190d4ae
openai/gpt-5.4-mini:none__json_schema PASS 100% 257 in → 24 out (0 reasoning)
ed18d3a
JSON Schema Structured Output
(raw) {"commit":"ed18d3a"}
openai/gpt-5.4-mini:xhigh PASS 100% 202 in → 88 out (79 reasoning)
7674313
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 249 in → 192 out (174 reasoning)
e029420
JSON Schema Structured Output
(raw) {"commit":"e029420"}
openai/gpt-5.4-nano:high PASS 100% 214 in → 77 out (66 reasoning)
85c55d1
openai/gpt-5.4-nano:high__json_schema PASS 100% 257 in → 81 out (61 reasoning)
7c3ffd6
JSON Schema Structured Output
(raw) {"commit":"7c3ffd6"}
openai/gpt-5.4-nano:low PASS 100% 214 in → 34 out (24 reasoning)
60fd69e
openai/gpt-5.4-nano:low__json_schema PASS 100% 257 in → 45 out (25 reasoning)
c1afac4
JSON Schema Structured Output
(raw) {"commit":"c1afac4"}
openai/gpt-5.4-nano:medium PASS 100% 214 in → 71 out (61 reasoning)
c8678ca
openai/gpt-5.4-nano:medium__json_schema PASS 100% 253 in → 83 out (65 reasoning)
460b966
JSON Schema Structured Output
(raw) {"commit":"460b966"}
openai/gpt-5.4-nano:none PASS 100% 220 in → 10 out (0 reasoning)
5c1fdd1
openai/gpt-5.4-nano:none__json_schema PASS 100% 249 in → 16 out (0 reasoning)
4777217
JSON Schema Structured Output
(raw) {"commit":"4777217"}
openai/gpt-5.4-nano:xhigh PASS 100% 222 in → 123 out (112 reasoning)
0f22aa8
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 261 in → 163 out (143 reasoning)
2f43ad5
JSON Schema Structured Output
(raw) {"commit":"2f43ad5"}
openai/gpt-5.4:high PASS 100% 216 in → 69 out (57 reasoning)
f4a4a45
openai/gpt-5.4:high__json_schema PASS 100% 261 in → 69 out (49 reasoning)
f47e91f
JSON Schema Structured Output
(raw) {"commit":"f47e91f"}
openai/gpt-5.4:low PASS 100% 208 in → 45 out (36 reasoning)
3379971
openai/gpt-5.4:low__json_schema PASS 100% 255 in → 37 out (18 reasoning)
90e4aad
JSON Schema Structured Output
(raw) {"commit":"90e4aad"}
openai/gpt-5.4:medium PASS 100% 210 in → 60 out (51 reasoning)
72af161
openai/gpt-5.4:medium__json_schema PASS 100% 253 in → 44 out (26 reasoning)
761d232
JSON Schema Structured Output
(raw) {"commit":"761d232"}
openai/gpt-5.4:none PASS 100% 214 in → 7 out (0 reasoning)
5865379
openai/gpt-5.4:none__json_schema PASS 100% 261 in → 16 out (0 reasoning)
bb05bef
JSON Schema Structured Output
(raw) {"commit":"bb05bef"}
openai/gpt-5.4:xhigh PASS 100% 212 in → 51 out (41 reasoning)
101c6bb
openai/gpt-5.4:xhigh__json_schema PASS 100% 255 in → 111 out (91 reasoning)
c9e645e
JSON Schema Structured Output
(raw) {"commit":"c9e645e"}
openai/gpt-5.5:high PASS 100% 206 in → 43 out (34 reasoning)
3700566
openai/gpt-5.5:high__json_schema PASS 100% 251 in → 16 out (0 reasoning)
181ae98
JSON Schema Structured Output
(raw) {"commit":"181ae98"}
openai/gpt-5.5:low PASS 100% 220 in → 23 out (12 reasoning)
23f7a58
openai/gpt-5.5:low__json_schema PASS 100% 257 in → 16 out (0 reasoning)
46345ef
JSON Schema Structured Output
(raw) {"commit":"46345ef"}
openai/gpt-5.5:medium PASS 100% 206 in → 24 out (15 reasoning)
d365ddd
openai/gpt-5.5:medium__json_schema PASS 100% 259 in → 19 out (0 reasoning)
eb5d8e6
JSON Schema Structured Output
(raw) {"commit":"eb5d8e6"}
openai/gpt-5.5:none PASS 100% 218 in → 9 out (0 reasoning)
e90d3ea
openai/gpt-5.5:none__json_schema PASS 100% 255 in → 19 out (0 reasoning)
f10f2e8
JSON Schema Structured Output
(raw) {"commit":"f10f2e8"}
openai/gpt-5.5:xhigh PASS 100% 212 in → 75 out (65 reasoning)
1ceb7cf
openai/gpt-5.5:xhigh__json_schema PASS 100% 253 in → 103 out (84 reasoning)
77f1093
JSON Schema Structured Output
(raw) {"commit":"77f1093"}
openai/gpt-5.6-luna:high PASS 100% 212 in → 7 out (0 reasoning)
1815531
openai/gpt-5.6-luna:high__json_schema PASS 100% 255 in → 17 out (0 reasoning)
57fed0f
JSON Schema Structured Output
(raw) {"commit":"57fed0f"}
openai/gpt-5.6-luna:low PASS 100% 210 in → 9 out (0 reasoning)
89a0d99
openai/gpt-5.6-luna:low__json_schema PASS 100% 251 in → 17 out (0 reasoning)
bec0fbd
JSON Schema Structured Output
(raw) {"commit":"bec0fbd"}
openai/gpt-5.6-luna:max PASS 100% 208 in → 8 out (0 reasoning)
c916a06
openai/gpt-5.6-luna:max__json_schema PASS 100% 259 in → 105 out (85 reasoning)
9bf31d2
JSON Schema Structured Output
(raw) {"commit":"9bf31d2"}
openai/gpt-5.6-luna:medium PASS 100% 212 in → 7 out (0 reasoning)
7982808
openai/gpt-5.6-luna:medium__json_schema PASS 100% 243 in → 16 out (0 reasoning)
f195199
JSON Schema Structured Output
(raw) {"commit":"f195199"}
openai/gpt-5.6-luna:none PASS 100% 210 in → 7 out (0 reasoning)
94fccde
openai/gpt-5.6-luna:none__json_schema PASS 100% 253 in → 17 out (0 reasoning)
68c776b
JSON Schema Structured Output
(raw) {"commit":"68c776b"}
openai/gpt-5.6-luna:xhigh PASS 100% 212 in → 8 out (0 reasoning)
3d56cbc
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 253 in → 17 out (0 reasoning)
f785d71
JSON Schema Structured Output
(raw) {"commit":"f785d71"}
openai/gpt-5.6-sol:high PASS 100% 216 in → 9 out (0 reasoning)
b38e3fc
openai/gpt-5.6-sol:high__json_schema PASS 100% 249 in → 16 out (0 reasoning)
785c548
JSON Schema Structured Output
(raw) {"commit":"785c548"}
openai/gpt-5.6-sol:low PASS 100% 210 in → 8 out (0 reasoning)
161e44d
openai/gpt-5.6-sol:low__json_schema PASS 100% 259 in → 18 out (0 reasoning)
bcbe8c7
JSON Schema Structured Output
(raw) {"commit":"bcbe8c7"}
openai/gpt-5.6-sol:max PASS 100% 210 in → 8 out (0 reasoning)
5ee0446
openai/gpt-5.6-sol:max__json_schema PASS 100% 255 in → 16 out (0 reasoning)
968e222
JSON Schema Structured Output
(raw) {"commit":"968e222"}
openai/gpt-5.6-sol:medium PASS 100% 212 in → 7 out (0 reasoning)
17fc202
openai/gpt-5.6-sol:medium__json_schema PASS 100% 251 in → 16 out (0 reasoning)
814fff4
JSON Schema Structured Output
(raw) {"commit":"814fff4"}
openai/gpt-5.6-sol:none PASS 100% 218 in → 10 out (0 reasoning)
3f2c00d
openai/gpt-5.6-sol:none__json_schema PASS 100% 253 in → 18 out (0 reasoning)
5e0e301
JSON Schema Structured Output
(raw) {"commit":"5e0e301"}
openai/gpt-5.6-sol:xhigh PASS 100% 206 in → 8 out (0 reasoning)
3db5fea
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 251 in → 17 out (0 reasoning)
06726d7
JSON Schema Structured Output
(raw) {"commit":"06726d7"}
openai/gpt-5.6-terra:high PASS 100% 216 in → 10 out (0 reasoning)
5c7cf9e
openai/gpt-5.6-terra:high__json_schema PASS 100% 255 in → 17 out (0 reasoning)
4ebed82
JSON Schema Structured Output
(raw) {"commit":"4ebed82"}
openai/gpt-5.6-terra:low PASS 100% 216 in → 8 out (0 reasoning)
30a4025
openai/gpt-5.6-terra:low__json_schema PASS 100% 257 in → 20 out (0 reasoning)
b3f8f3e
JSON Schema Structured Output
(raw) {"commit":"b3f8f3e"}
openai/gpt-5.6-terra:max PASS 100% 214 in → 9 out (0 reasoning)
0e0861f
openai/gpt-5.6-terra:max__json_schema PASS 100% 253 in → 18 out (0 reasoning)
1d9349f
JSON Schema Structured Output
(raw) {"commit":"1d9349f"}
openai/gpt-5.6-terra:medium PASS 100% 212 in → 8 out (0 reasoning)
864ca6e
openai/gpt-5.6-terra:medium__json_schema PASS 100% 253 in → 18 out (0 reasoning)
8e5657c
JSON Schema Structured Output
(raw) {"commit":"8e5657c"}
openai/gpt-5.6-terra:none PASS 100% 216 in → 9 out (0 reasoning)
c5a1279
openai/gpt-5.6-terra:none__json_schema PASS 100% 253 in → 18 out (0 reasoning)
b160c9b
JSON Schema Structured Output
(raw) {"commit":"b160c9b"}
openai/gpt-5.6-terra:xhigh PASS 100% 210 in → 8 out (0 reasoning)
ae2a883
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 253 in → 19 out (0 reasoning)
1c4aa1f
JSON Schema Structured Output
(raw) {"commit":"1c4aa1f"}
openai/gpt-oss-120b:high PASS 100% 273 in → 257 out (237 reasoning)
0d31c2b (change port)
openai/gpt-oss-120b:high__json_schema PASS 100% 275 in → 504 out (484 reasoning)
bdc21e2
JSON Schema Structured Output
(raw) { "commit": "bdc21e2" }
openai/gpt-oss-120b:low PASS 100% 269 in → 77 out (60 reasoning)
a827ea7 (change port)
openai/gpt-oss-120b:low__json_schema PASS 100% 271 in → 68 out (50 reasoning)
81d3552
JSON Schema Structured Output
(raw) {"commit":"81d3552"}
openai/gpt-oss-120b:medium PASS 100% 266 in → 128 out (97 reasoning)
277f777 (change port)
openai/gpt-oss-120b:medium__json_schema PASS 100% 354 in → 96 out (0 reasoning)
a45c0c8
JSON Schema Structured Output
(raw) { "commit": "a45c0c8" }
openai/gpt-oss-120b:xhigh PASS 100% 271 in → 266 out (247 reasoning)
fe46e3c (change port)
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 285 in → 282 out (246 reasoning)
3d2f5a0
JSON Schema Structured Output
(raw) { "commit": "3d2f5a0" }
openai/gpt-oss-20b:high PASS 100% 276 in → 586 out (551 reasoning)
The failing commit is **1e25259** – the “change port” commit introduced the bug.
openai/gpt-oss-20b:high__json_schema PASS 100% 273 in → 678 out (728 reasoning)
090cebf
JSON Schema Structured Output
(raw) {"commit":"090cebf"}
openai/gpt-oss-20b:low PASS 100% 260 in → 69 out (54 reasoning)
ca4384c
openai/gpt-oss-20b:low__json_schema PASS 100% 279 in → 42 out (19 reasoning)
5a745a6
JSON Schema Structured Output
(raw) { "commit": "5a745a6" }
openai/gpt-oss-20b:medium PASS 100% 271 in → 216 out (185 reasoning)
5bd29cc (The `"change port"` commit is the first one that causes the tests to fail.)
openai/gpt-oss-20b:medium__json_schema PASS 100% 271 in → 172 out (168 reasoning)
b1f6948
JSON Schema Structured Output
(raw) { "commit": "b1f6948" }
openai/gpt-oss-20b:xhigh PASS 100% 271 in → 384 out (342 reasoning)
f5d0556 (the “change port” commit)
poolside/laguna-m.1:high PASS 100% 244 in → 665 out (655 reasoning)
c089201
poolside/laguna-m.1:high__json_schema PASS 100% 252 in → 464 out (450 reasoning)
630b385
JSON Schema Structured Output
(raw) {"commit": "630b385"}
poolside/laguna-m.1:low PASS 100% 250 in → 652 out (642 reasoning)
f9e5936
poolside/laguna-m.1:low__json_schema PASS 100% 254 in → 421 out (403 reasoning)
4065975
JSON Schema Structured Output
(raw) { "commit": "4065975" }
poolside/laguna-m.1:medium PASS 100% 252 in → 373 out (364 reasoning)
c7aa663
poolside/laguna-m.1:medium__json_schema PASS 100% 246 in → 495 out (483 reasoning)
4635bcd
JSON Schema Structured Output
(raw) {"commit": "4635bcd"}
poolside/laguna-m.1:none PASS 100% 252 in → 9 out (0 reasoning)
f483a82
poolside/laguna-m.1:none__json_schema PASS 100% 252 in → 13 out (0 reasoning)
9e6c431
JSON Schema Structured Output
(raw) {"commit": "9e6c431"}
poolside/laguna-m.1:xhigh PASS 100% 250 in → 426 out (416 reasoning)
e9078d6
poolside/laguna-m.1:xhigh__json_schema PASS 100% 252 in → 560 out (546 reasoning)
2224e89
JSON Schema Structured Output
(raw) {"commit": "2224e89"}
poolside/laguna-xs-2.1:high PASS 100% 246 in → 363 out (256 reasoning)
The bad commit is c39c7b5. **Reasoning:** - Tests passed for commits up to `64cfbfc` (add default config). - The first failure occurred at `c39c7b5` (change port). - Subsequent commits (`a7392e4`, `b6512db`) inherit the broken state, but the root cause is the first failing commit. Answer: c39c7b5
poolside/laguna-xs-2.1:high__json_schema PASS 100% 252 in → 479 out (460 reasoning)
c3b1c4b
JSON Schema Structured Output
(raw) { "commit": "c3b1c4b" }
poolside/laguna-xs-2.1:low PASS 100% 252 in → 773 out (709 reasoning)
The bad commit is the first one that caused the test to fail. The test passed for the initial commit and adding the default config, but failed when the port was changed. Subsequent commits also failed, so the root cause is the first failure point. **Answer:** 541e4f1
poolside/laguna-xs-2.1:low__json_schema PASS 100% 248 in → 498 out (485 reasoning)
65ed133
JSON Schema Structured Output
(raw) {"commit": "65ed133"}
poolside/laguna-xs-2.1:medium PASS 100% 250 in → 637 out (499 reasoning)
The bad commit is **6c1830a**. **Reasoning:** The test results show that the build passes for commits up to `5f75dcf` (add default config). The first failure occurs at `6c1830a` (change port), which introduces the bug. Subsequent commits (`63d57e1` and `ad2fe17`) inherit the failure but are not the root cause. The commit subject line for `6c1830a` is "change port", but the hash is the direct identifier. **Answer:** 6c1830a
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 248 in → 417 out (402 reasoning)
2dfa2ae
JSON Schema Structured Output
(raw) { "commit": "2dfa2ae" }
poolside/laguna-xs-2.1:none PASS 100% 248 in → 5 out (0 reasoning)
4bdf8a2
poolside/laguna-xs-2.1:none__json_schema PASS 100% 254 in → 16 out (0 reasoning)
0f7a8ea
JSON Schema Structured Output
(raw) { "commit": "0f7a8ea" }
poolside/laguna-xs-2.1:xhigh PASS 100% 254 in → 523 out (396 reasoning)
The bad commit is f984149. **Explanation:** - The test results show that the first failure occurs at commit `f984149` (change port). - All subsequent commits build on top of this commit and inherit the failure. - Since the commits before `f984149` (`6751886` and `ed833b4`) passed, the bug must have been introduced in `f984149`. Answer: `f984149`
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 244 in → 791 out (778 reasoning)
3fc937a
JSON Schema Structured Output
(raw) {"commit": "3fc937a"}
poolside/laguna-xs.2:high PASS 100% 281 in → 346 out (249 reasoning)
Looking at the test results chronologically: - 6b98fd7 Initial commit: PASS - aac878b add default config: PASS - 2dcacd9 change port: **FAIL** ← First failure The tests pass until commit `2dcacd9` ("change port"), which is the first commit to fail. This is the bad commit that introduced the bug. **Answer: 2dcacd9**
poolside/laguna-xs.2:high__json_schema PASS 100% 287 in → 282 out (264 reasoning)
708b7ef
JSON Schema Structured Output
(raw) { "commit": "708b7ef" }
poolside/laguna-xs.2:low PASS 100% 287 in → 218 out (160 reasoning)
Based on the test results, the first commit where the build fails is **f1530aa** with the subject "change port". This is the commit that introduced the bug, as all subsequent commits also fail. **Answer:** f1530aa
poolside/laguna-xs.2:low__json_schema PASS 100% 289 in → 295 out (283 reasoning)
410ca02
JSON Schema Structured Output
(raw) {"commit":"410ca02"}
poolside/laguna-xs.2:medium PASS 100% 293 in → 385 out (265 reasoning)
Based on the test results, I can see that: 1. **b76f4f4 Initial commit**: PASS 2. **7d9e007 add default config**: PASS 3. **880a821 change port**: FAIL ← First failure The build was passing up to and including commit `7d9e007`, but then failed starting with commit `880a821`. This indicates that commit introduced the bug. **Bad commit: 880a821**
poolside/laguna-xs.2:medium__json_schema PASS 100% 293 in → 340 out (327 reasoning)
533e500
JSON Schema Structured Output
(raw) {"commit":"533e500"}
poolside/laguna-xs.2:none PASS 100% 289 in → 74 out (0 reasoning)
Looking at the test results, I can see that: - Commits `4cc8e71` and `050fb45` pass - Commit `8f190a8` fails and all subsequent commits also fail The bad commit is `8f190a8` with the subject "change port".
poolside/laguna-xs.2:none__json_schema PASS 100% 293 in → 15 out (0 reasoning)
784e56e change port
JSON Schema Structured Output
(raw) {"commit": "784e56e change port"}
poolside/laguna-xs.2:xhigh PASS 100% 287 in → 403 out (291 reasoning)
Looking at the test results chronologically: - b6ebc46 Initial commit: PASS - c2f9ff2 add default config: PASS - e063001 change port: FAIL The tests pass up until commit `c2f9ff2`, but fail starting with `e063001`. This means the bug was introduced in commit `e063001` with the subject "change port". **Answer: e063001**
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 287 in → 290 out (271 reasoning)
4906688
JSON Schema Structured Output
(raw) { "commit": "4906688" }
qwen/qwen3.6-27b:high PASS 100% 244 in → 470 out (357 reasoning)
41138d8
qwen/qwen3.6-27b:high__json_schema PASS 100% 250 in → 593 out (1 reasoning)
6a178d4
JSON Schema Structured Output
(raw) { "commit": "6a178d4" }
qwen/qwen3.6-27b:low PASS 100% 246 in → 1,097 out (826 reasoning)
dd5dd3b
qwen/qwen3.6-27b:low__json_schema PASS 100% 244 in → 547 out (449 reasoning)
aca61df
JSON Schema Structured Output
(raw) { "commit": "aca61df" }
qwen/qwen3.6-27b:medium PASS 100% 246 in → 705 out (549 reasoning)
60b2797
qwen/qwen3.6-27b:medium__json_schema PASS 100% 240 in → 373 out (1 reasoning)
d120fad change port
JSON Schema Structured Output
(raw) { "commit": "d120fad change port" }
qwen/qwen3.6-27b:none PASS 100% 250 in → 7 out (0 reasoning)
bb7c447
qwen/qwen3.6-27b:none__json_schema PASS 100% 242 in → 20 out (0 reasoning)
296d80b change port
JSON Schema Structured Output
(raw) { "commit": "296d80b change port" }
qwen/qwen3.6-27b:xhigh PASS 100% 254 in → 802 out (605 reasoning)
399016b
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 246 in → 490 out (398 reasoning)
07e75af
JSON Schema Structured Output
(raw) { "commit": "07e75af" }
qwen/qwen3.6-35b-a3b:high PASS 100% 246 in → 454 out (367 reasoning)
8e75d5f
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 252 in → 1,065 out (809 reasoning)
5221868
JSON Schema Structured Output
(raw) { "commit": "5221868" }
qwen/qwen3.6-35b-a3b:low PASS 100% 252 in → 579 out (450 reasoning)
27ed23d
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 242 in → 705 out (548 reasoning)
f2b0fcd
JSON Schema Structured Output
(raw) { "commit": "f2b0fcd" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 246 in → 945 out (722 reasoning)
5e915a0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 248 in → 751 out (581 reasoning)
6b0b8c9
JSON Schema Structured Output
(raw) { "commit": "6b0b8c9" }
qwen/qwen3.6-35b-a3b:none PASS 100% 250 in → 6 out (0 reasoning)
ce53469
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 252 in → 18 out (0 reasoning)
6c4e0d9
JSON Schema Structured Output
(raw) { "commit": "6c4e0d9" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 238 in → 380 out (326 reasoning)
9727bee
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 252 in → 472 out (326 reasoning)
0c0e1e8
JSON Schema Structured Output
(raw) { "commit": "0c0e1e8" }
qwen/qwen3.6-flash:high PASS 100% 252 in → 1,063 out (1,051 reasoning)
366b04f
qwen/qwen3.6-flash:low PASS 100% 248 in → 377 out (365 reasoning)
e4b2a71
qwen/qwen3.6-flash:medium PASS 100% 238 in → 383 out (373 reasoning)
524aba2
qwen/qwen3.6-flash:none PASS 100% 248 in → 6 out (0 reasoning)
5727ef2
qwen/qwen3.6-flash:xhigh PASS 100% 252 in → 569 out (557 reasoning)
815c966
qwen/qwen3.7-flash:high PASS 100% 248 in → 339 out (327 reasoning)
c654954
qwen/qwen3.7-flash:low PASS 100% 242 in → 307 out (296 reasoning)
fbd6d1b
qwen/qwen3.7-flash:max PASS 100% 248 in → 340 out (328 reasoning)
455196b
qwen/qwen3.7-flash:medium PASS 100% 244 in → 343 out (333 reasoning)
1b3eba4
qwen/qwen3.7-flash:none PASS 100% 246 in → 6 out (0 reasoning)
7eb6011
qwen/qwen3.7-flash:xhigh PASS 100% 244 in → 464 out (453 reasoning)
06f39ea
qwen/qwen3.7-max:high PASS 100% 248 in → 206 out (195 reasoning)
aa610f7
qwen/qwen3.7-max:low PASS 100% 246 in → 202 out (191 reasoning)
000ab48
qwen/qwen3.7-max:medium PASS 100% 246 in → 150 out (140 reasoning)
49eceb7
qwen/qwen3.7-max:none PASS 100% 248 in → 75 out (0 reasoning)
Based on the test results, the build was passing at commit `057a7ef` and started failing at commit `8b43555`. Since the tests are ordered oldest first and the first failure occurs at `8b43555`, this is the commit that introduced the bug. **8b43555**
qwen/qwen3.7-max:xhigh PASS 100% 246 in → 203 out (192 reasoning)
1e4f5bc
qwen/qwen3.7-plus:high PASS 100% 250 in → 261 out (191 reasoning)
The bad commit is **88a0e8f** ("change port"). This is the first commit where the test fails. The previous commit `067279f` ("add default config") passed, so `88a0e8f` is the commit that introduced the bug.
qwen/qwen3.7-plus:low PASS 100% 250 in → 288 out (184 reasoning)
The bad commit is **b2c205a** (subject: "change port"). Reasoning: This is the first commit in the history (from oldest to newest) where the test result transitions from PASS to FAIL. The previous commit `383caed add default config` passed, and the very next commit `b2c205a change port` failed. This makes `b2c205a` the commit that introduced the bug.
qwen/qwen3.7-plus:medium PASS 100% 246 in → 267 out (190 reasoning)
The bad commit is **c70f1fe** with the subject line **"change port"**. This is the first commit where the test result changed from PASS to FAIL. The previous commit (`17bb20f add default config`) passed, so `c70f1fe change port` is the commit that introduced the bug.
qwen/qwen3.7-plus:none PASS 100% 248 in → 5 out (0 reasoning)
d0fec4c
qwen/qwen3.7-plus:xhigh PASS 100% 244 in → 365 out (170 reasoning)
The bad commit is **cfe90e3** ("change port"). **Reasoning:** This is a straightforward git bisect scenario. Looking at the test results from oldest to newest: - `a12ed55` Initial commit: **PASS** ✅ - `ecc82d2` add default config: **PASS** ✅ - `cfe90e3` change port: **FAIL** ❌ - `891b912` add feature flag: **FAIL** ❌ - `6fd9c36` add port constant: **FAIL** ❌ The last passing commit is `ecc82d2`, and the first failing commit is `cfe90e3`. Therefore, `cfe90e3` introduced the bug. All subsequent commits continue to fail, consistent with the build remaining broken from that point forward.
tencent/hy3:high PASS 100% 216 in → 445 out (438 reasoning)
1d68c58
tencent/hy3:high__json_schema PASS 100% 218 in → 328 out (284 reasoning)
2452086
JSON Schema Structured Output
(raw) { "commit": "2452086" }
tencent/hy3:low PASS 100% 220 in → 516 out (510 reasoning)
5abb60b
tencent/hy3:low__json_schema PASS 100% 214 in → 370 out (313 reasoning)
057d69d
JSON Schema Structured Output
(raw) { "commit": "057d69d" }
tencent/hy3:medium PASS 100% 222 in → 437 out (430 reasoning)
a98e5ca
tencent/hy3:medium__json_schema PASS 100% 222 in → 386 out (334 reasoning)
2c87420
JSON Schema Structured Output
(raw) { "commit": "2c87420" }
tencent/hy3:none PASS 100% 223 in → 7 out (0 reasoning)
47c625e change port
tencent/hy3:none__json_schema PASS 100% 227 in → 13 out (0 reasoning)
22f2ad8
JSON Schema Structured Output
(raw) { "commit": "22f2ad8" }
tencent/hy3:xhigh PASS 100% 218 in → 501 out (495 reasoning)
56e76ca
tencent/hy3:xhigh__json_schema PASS 100% 220 in → 544 out (502 reasoning)
0242c62
JSON Schema Structured Output
(raw) { "commit": "0242c62" }
thinkingmachines/inkling-small:high PASS 100% 221 in → 29 out (19 reasoning)
d302cfa
thinkingmachines/inkling-small:high__json_schema PASS 100% 217 in → 134 out (120 reasoning)
4254e11
JSON Schema Structured Output
(raw) {"commit": "4254e11"}
thinkingmachines/inkling-small:low PASS 100% 231 in → 28 out (11 reasoning)
b4d114d (change port)
thinkingmachines/inkling-small:low__json_schema PASS 100% 223 in → 27 out (13 reasoning)
f68ae59
JSON Schema Structured Output
(raw) {"commit": "f68ae59"}
thinkingmachines/inkling-small:max PASS 100% 223 in → 47 out (36 reasoning)
b1e7887
thinkingmachines/inkling-small:max__json_schema PASS 100% 217 in → 37 out (20 reasoning)
c89dd7b
JSON Schema Structured Output
(raw) { "commit": "c89dd7b" }
thinkingmachines/inkling-small:medium PASS 100% 217 in → 27 out (18 reasoning)
619116a
thinkingmachines/inkling-small:medium__json_schema PASS 100% 217 in → 37 out (20 reasoning)
1937b97
JSON Schema Structured Output
(raw) { "commit": "1937b97" }
thinkingmachines/inkling-small:none PASS 100% 219 in → 6 out (0 reasoning)
6eff949<|end_message|>
thinkingmachines/inkling-small:xhigh PASS 100% 229 in → 73 out (62 reasoning)
f8a424b
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 221 in → 45 out (33 reasoning)
71fe270
JSON Schema Structured Output
(raw) {"commit":"71fe270"}
thinkingmachines/inkling:high PASS 100% 223 in → 129 out (94 reasoning)
f9c6ac8
thinkingmachines/inkling:low PASS 100% 221 in → 24 out (13 reasoning)
d59e578
thinkingmachines/inkling:low__json_schema PASS 100% 221 in → 38 out (19 reasoning)
0d36e7e
JSON Schema Structured Output
(raw) { "commit": "0d36e7e" }
thinkingmachines/inkling:max PASS 100% 221 in → 203 out (120 reasoning)
The first commit that fails is **486146c** (subject: `change port`). - `bd7c1d7` PASS - `945d5a6` PASS - `486146c` FAIL ← bug introduced here - `6675dbe` FAIL - `aaee787` FAIL **Answer:** `486146c`
thinkingmachines/inkling:max__json_schema PASS 100% 221 in → 138 out (122 reasoning)
cfbadb5
JSON Schema Structured Output
(raw) { "commit": "cfbadb5" }
thinkingmachines/inkling:medium PASS 100% 225 in → 81 out (70 reasoning)
83da9d9
thinkingmachines/inkling:medium__json_schema PASS 100% 225 in → 184 out (168 reasoning)
61180b6
JSON Schema Structured Output
(raw) { "commit": "61180b6" }
thinkingmachines/inkling:none PASS 100% 225 in → 7 out (0 reasoning)
65c7674<|end_message|>
thinkingmachines/inkling:none__json_schema PASS 100% 227 in → 9 out (0 reasoning)
fa25243
JSON Schema Structured Output
(raw) {"commit":"fa25243"}
thinkingmachines/inkling:xhigh PASS 100% 227 in → 134 out (94 reasoning)
2732c8b
x-ai/grok-4.3:high PASS 100% 389 in → 355 out (351 reasoning)
57b0656
x-ai/grok-4.3:high__json_schema PASS 100% 456 in → 436 out (427 reasoning)
ab68d9d
JSON Schema Structured Output
(raw) {"commit":"ab68d9d"}
x-ai/grok-4.3:low PASS 100% 401 in → 434 out (341 reasoning)
The test results show that commits 61ffcf6 and 5ed0c1f both pass. The very next commit in the history (541e4f1) is the first one to produce a failing test. All later commits continue to fail, confirming that the bug was introduced exactly when "change port" landed and was never subsequently fixed. Thus the bad commit is 541e4f1. Final answer: 541e4f1
x-ai/grok-4.3:low__json_schema PASS 100% 454 in → 338 out (329 reasoning)
ca955c3
JSON Schema Structured Output
(raw) {"commit": "ca955c3"}
x-ai/grok-4.3:max PASS 100% 391 in → 463 out (460 reasoning)
b869933
x-ai/grok-4.3:max__json_schema PASS 100% 448 in → 398 out (391 reasoning)
c158fff
JSON Schema Structured Output
(raw) {"commit":"c158fff"}
x-ai/grok-4.3:medium PASS 100% 401 in → 624 out (520 reasoning)
The commit history is linear, and the provided test results explicitly show a clean pass/fail transition point. The two oldest commits both exit with code 0 (green builds). The third commit, `1f21d76`, is the first one whose test run exits with a non-zero code; every subsequent commit also fails. Consequently the single commit that introduced the regression is the one whose subject line is “change port” (hash `1f21d76`). Final answer: 1f21d76
x-ai/grok-4.3:medium__json_schema PASS 100% 456 in → 451 out (443 reasoning)
cdfa4da
JSON Schema Structured Output
(raw) {"commit":"cdfa4da"}
x-ai/grok-4.3:none PASS 100% 387 in → 4 out (0 reasoning)
f389cb4
x-ai/grok-4.3:none__json_schema PASS 100% 444 in → 8 out (0 reasoning)
0aaf71b
JSON Schema Structured Output
(raw) {"commit":"0aaf71b"}
x-ai/grok-4.3:xhigh PASS 100% 393 in → 391 out (386 reasoning)
0eb7fd4
x-ai/grok-4.3:xhigh__json_schema PASS 100% 448 in → 619 out (610 reasoning)
afd5cf4
JSON Schema Structured Output
(raw) {"commit": "afd5cf4"}
x-ai/grok-4.5:high PASS 100% 413 in → 259 out (256 reasoning)
803deba
x-ai/grok-4.5:high__json_schema PASS 100% 494 in → 302 out (294 reasoning)
6324657
JSON Schema Structured Output
(raw) {"commit":"6324657"}
x-ai/grok-4.5:low PASS 100% 419 in → 182 out (177 reasoning)
2519d5e
x-ai/grok-4.5:low__json_schema PASS 100% 486 in → 205 out (195 reasoning)
0d3673f
JSON Schema Structured Output
(raw) {"commit":"0d3673f"}
x-ai/grok-4.5:max PASS 100% 405 in → 277 out (274 reasoning)
623ddab
x-ai/grok-4.5:max__json_schema PASS 100% 480 in → 207 out (199 reasoning)
4529337
JSON Schema Structured Output
(raw) {"commit":"4529337"}
x-ai/grok-4.5:medium PASS 100% 413 in → 251 out (247 reasoning)
87f1061
x-ai/grok-4.5:medium__json_schema PASS 100% 490 in → 286 out (274 reasoning)
f52773d
JSON Schema Structured Output
(raw) { "commit": "f52773d" }
x-ai/grok-4.5:xhigh PASS 100% 413 in → 235 out (231 reasoning)
e78a313
x-ai/grok-4.5:xhigh__json_schema PASS 100% 484 in → 305 out (297 reasoning)
36aa612
JSON Schema Structured Output
(raw) {"commit":"36aa612"}
z-ai/glm-4.7-flash:high PASS 100% 213 in → 650 out (527 reasoning)
e814324
z-ai/glm-4.7-flash:high__json_schema PASS 100% 215 in → 664 out (581 reasoning)
be869f5
JSON Schema Structured Output
(raw) { "commit": "be869f5" }
z-ai/glm-4.7-flash:low PASS 100% 217 in → 720 out (610 reasoning)
bf7a483
z-ai/glm-4.7-flash:low__json_schema PASS 100% 221 in → 551 out (456 reasoning)
07d8c16
JSON Schema Structured Output
(raw) { "commit": "07d8c16" }
z-ai/glm-4.7-flash:medium PASS 100% 213 in → 631 out (540 reasoning)
d1cef2e
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 221 in → 730 out (582 reasoning)
3b75114
JSON Schema Structured Output
(raw) { "commit": "3b75114" }
z-ai/glm-4.7-flash:none PASS 100% 217 in → 7 out (0 reasoning)
397c869 change port
z-ai/glm-4.7-flash:xhigh PASS 100% 219 in → 762 out (668 reasoning)
8c19e61
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 221 in → 632 out (480 reasoning)
b3f5860
JSON Schema Structured Output
(raw) { "commit": "b3f5860" }
z-ai/glm-5.2:high PASS 100% 218 in → 195 out (177 reasoning)
36525e6
z-ai/glm-5.2:high__json_schema PASS 100% 222 in → 228 out (214 reasoning)
5fe012a
JSON Schema Structured Output
(raw) { "commit": "5fe012a" }
z-ai/glm-5.2:low PASS 100% 228 in → 129 out (121 reasoning)
6e22d3c
z-ai/glm-5.2:low__json_schema PASS 100% 223 in → 256 out (193 reasoning)
5d2f9a9
JSON Schema Structured Output
(raw) { "commit": "5d2f9a9" }
z-ai/glm-5.2:medium PASS 100% 224 in → 214 out (170 reasoning)
The bad commit that introduced the bug is: 53c9a7e
z-ai/glm-5.2:medium__json_schema PASS 100% 232 in → 110 out (77 reasoning)
a5be1c6
JSON Schema Structured Output
(raw) { "commit": "a5be1c6" }
z-ai/glm-5.2:none PASS 100% 218 in → 6 out (0 reasoning)
8e9ddd8
z-ai/glm-5.2:none__json_schema PASS 100% 220 in → 13 out (0 reasoning)
1fe87f8 change port
JSON Schema Structured Output
(raw) {"commit": "1fe87f8 change port"}
z-ai/glm-5.2:xhigh PASS 100% 220 in → 104 out (90 reasoning)
868e64f
z-ai/glm-5.2:xhigh__json_schema PASS 100% 228 in → 188 out (174 reasoning)
cd0fba6
JSON Schema Structured Output
(raw) { "commit": "cd0fba6" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid structured output. Output: 8414171
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid JSON. Output: The bad commit is **e3083c3** — it's the first commit after "add default config" that caused the build to fail, as subsequent commits also failed but only after this point. **Answer:** e3083c3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 271 in → 7 out (0 reasoning)
analysis
JSON Schema Structured Output
(raw) {"commit":"analysis"}
Failure: Expected target hash/prefix or subject 'change port'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aa5dc41a-2035-986a-8fe0-d4255f149841","request_id":"aa5dc41a-2035-986a-8fe0-d4255f149841"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ad5f940e-67ff-9c73-8409-315aac548073","request_id":"ad5f940e-67ff-9c73-8409-315aac548073"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d5460bad-3648-9dd2-bbe5-69ae324faac7","request_id":"d5460bad-3648-9dd2-bbe5-69ae324faac7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-82bfe362-06df-961e-a7f3-207e0df6bf86","request_id":"82bfe362-06df-961e-a7f3-207e0df6bf86"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-879274c3-fbbb-9fa3-8081-a1dab5b409ec","request_id":"879274c3-fbbb-9fa3-8081-a1dab5b409ec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-869098e4-5a45-9929-9d65-0fe82a7694ff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a9aee63d-cf1b-95ad-960b-6f7094ac1d7b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-97c16312-90ad-93af-b76c-5dd021be3966"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5f4090b0-f124-98f3-b301-e35ff0f7be61"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6d061d49-e9b5-91f6-b225-c3700e6b6aac"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-20fb7831-4702-964c-8cc9-922d1585b571"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-afd19ae5-3a21-95d5-b073-58d73cafdf3b","request_id":"afd19ae5-3a21-95d5-b073-58d73cafdf3b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c31252df-392d-9bba-846a-4112c879d81a","request_id":"c31252df-392d-9bba-846a-4112c879d81a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0831c242-630e-9114-bbb6-3cc01a81e5e8","request_id":"0831c242-630e-9114-bbb6-3cc01a81e5e8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-01fb5fb4-7fe3-98c2-b836-316c6fbdb2b1","request_id":"01fb5fb4-7fe3-98c2-b836-316c6fbdb2b1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9c84996b-5c63-9d04-a39b-b951ec86b00c","request_id":"9c84996b-5c63-9d04-a39b-b951ec86b00c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6b6ccfe2-4d6f-92db-b866-3163ff8db948","request_id":"6b6ccfe2-4d6f-92db-b866-3163ff8db948"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-94c1e47d-4e14-9360-bacd-21d230c5f7ee","request_id":"94c1e47d-4e14-9360-bacd-21d230c5f7ee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4f06493c-fe82-9bc7-baad-e3b451326245","request_id":"4f06493c-fe82-9bc7-baad-e3b451326245"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e10b4340-932b-9390-a5c2-9f13fb958414","request_id":"e10b4340-932b-9390-a5c2-9f13fb958414"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6e2998d0-c1a4-9dfd-914a-6a23a5e99e23","request_id":"6e2998d0-c1a4-9dfd-914a-6a23a5e99e23"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: 1d40860{"commit":"1d40860"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: The first failing commit is **b4a978b** (subject: "change port"). Tests pass up through `b81afb0`, then fail starting at `b4a978b`, so that commit introduced the bug.
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: The build first fails at **ba7133d** (`change port`). **Answer:** `ba7133d` (or `change port`)
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 219 in → 10 out (0 reasoning)
39dc92f
JSON Schema Structured Output
(raw) {"commit": "39dc92f"}
Failure: Expected target hash/prefix or subject 'change port'