Branching history with bad commit that removes validation - model identifies the bad commit
Tests ability to identify a commit that removed validation logic via bisect. Evaluates bisect for a regression that removes safety checks.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo '#!/bin/bash' > test.sh
  5. 05 echo 'grep -q "if not data" src/auth.py' >> test.sh
  6. 06 chmod +x test.sh
  7. 07 mkdir -p src && echo 'def validate_input(data): if not data: return False return True' > src/auth.py
  8. 08 git add test.sh src/auth.py && git commit -m 'Initial commit'
  9. 09 echo '# validation enabled' >> src/auth.py
  10. 10 git add src/auth.py && git commit -m 'add validation function'
  11. 11 echo 'def validate_input(data): return True' > src/auth.py
  12. 12 git add src/auth.py && git commit -m 'remove validation logic'
  13. 13 git checkout -b feature
  14. 14 echo '# auth logging enabled' >> src/auth.py
  15. 15 git add src/auth.py && git commit -m 'add auth logging'
  16. 16 git checkout main
  17. 17 git merge feature
Prompt
You are debugging a git repository. The following commit history has one bad commit that broke the build. Identify the bad commit hash (first 7 characters) or commit subject line that introduced the bug. Use the git log and test output to determine which commit is bad. Your answer should be the commit hash (e.g., 'abc1234') or the commit subject line.
Expected
remove validation logic

Scoped model quality, cost, API time, and token usage for git_bisect/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 275 in → 202 out (22 reasoning)
The bad commit is **449cc95** — "remove validation logic" **Reasoning:** - `f7b889f` (Initial commit): PASS - `5a9475a` (add validation function): PASS - `449cc95` (remove validation logic): **FAIL** ← first failing commit - `e59cf65` (add auth logging): FAIL (inherited the breakage) The build was passing until `449cc95`, which removed validation logic that the code (or tests) still depended on, breaking the build. All subsequent commits fail as a result.
anthropic/claude-fable-5:high__json_schema PASS 100% 488 in → 14 out (0 reasoning)
e04899e
JSON Schema Structured Output
(raw) {"commit": "e04899e"}
anthropic/claude-fable-5:low PASS 100% 279 in → 44 out (0 reasoning)
18c3dcb — "remove validation logic" is the bad commit (first failing commit; tests passed before it and fail from it onward).
anthropic/claude-fable-5:low__json_schema PASS 100% 492 in → 14 out (0 reasoning)
e42984e
JSON Schema Structured Output
(raw) {"commit": "e42984e"}
anthropic/claude-fable-5:max PASS 100% 275 in → 478 out (74 reasoning)
**Bad commit: `3be9fc1` — "remove validation logic"** **Reasoning (bisect logic):** | Commit | Subject | Result | |--------|---------|--------| | b551aeb | Initial commit | ✅ PASS | | 55101c4 | add validation function | ✅ PASS | | **3be9fc1** | **remove validation logic** | ❌ **FAIL** ← first failure | | 4926830 | add auth logging | ❌ FAIL | The build passed through `55101c4` and first failed at `3be9fc1`. Since every commit before it is good and every commit from that point onward is bad, `3be9fc1` ("remove validation logic") is the commit that introduced the bug. The subsequent commit `4926830` fails only because it's built on top of the broken change. The subject line also supports this: removing validation logic that a prior commit added likely broke tests that depended on it.
anthropic/claude-fable-5:max__json_schema PASS 100% 504 in → 131 out (57 reasoning)
5baf178
JSON Schema Structured Output
(raw) {"commit": "5baf178"}
anthropic/claude-fable-5:medium PASS 100% 283 in → 79 out (21 reasoning)
**3952ba1** — "remove validation logic" The tests pass through `1b0d59c` and first fail at `3952ba1`, so that commit introduced the break (likely removing validation logic that other code still depended on).
anthropic/claude-fable-5:medium__json_schema PASS 100% 494 in → 14 out (0 reasoning)
0536bea
JSON Schema Structured Output
(raw) {"commit": "0536bea"}
anthropic/claude-fable-5:xhigh PASS 100% 267 in → 302 out (18 reasoning)
**Bad commit: `9e41897` — "remove validation logic"** **Reasoning (bisect logic):** | Commit | Subject | Test Result | |--------|---------|-------------| | e12374c | Initial commit | ✅ PASS | | 683f551 | add validation function | ✅ PASS | | 9e41897 | remove validation logic | ❌ FAIL ← first failure | | 2178436 | add auth logging | ❌ FAIL | The tests pass through `683f551` and first fail at `9e41897`. Since the build was healthy immediately before and broken immediately after, `9e41897` ("remove validation logic") is the commit that introduced the bug — removing the validation logic evidently broke functionality that the tests depend on. The subsequent commit (`2178436`) just inherits the broken state.
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 494 in → 15 out (0 reasoning)
2d70a15
JSON Schema Structured Output
(raw) {"commit": "2d70a15"}
anthropic/claude-haiku-4.5:high PASS 100% 242 in → 483 out (286 reasoning)
# Analysis Looking at the test results in chronological order: 1. **98b40bc** (Initial commit): PASS ✓ 2. **eabe4f7** (add validation function): PASS ✓ 3. **6ef1fe3** (remove validation logic): FAIL ✗ 4. **7330b96** (add auth logging): FAIL ✗ The build transitioned from PASS to FAIL at commit **6ef1fe3**. This is where the validation logic was removed, which broke the build. **Answer: `6ef1fe3`** or **`remove validation logic`**
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 406 in → 375 out (328 reasoning)
c4dd7cd
JSON Schema Structured Output
(raw) {"commit": "c4dd7cd"}
anthropic/claude-haiku-4.5:low PASS 100% 240 in → 459 out (253 reasoning)
# Analysis Looking at the test results in chronological order: 1. 8858986 (Initial commit): **PASS** ✓ 2. 8dbb522 (add validation function): **PASS** ✓ 3. 5c47e2d (remove validation logic): **FAIL** ✗ ← First failure 4. 45ebcd5 (add auth logging): **FAIL** ✗ The build passes through commit 8dbb522, but fails starting with commit 5c47e2d. This is the first commit where tests fail. **Answer: `5c47e2d`** (or "remove validation logic")
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 408 in → 359 out (323 reasoning)
211b03d
JSON Schema Structured Output
(raw) {"commit": "211b03d"}
anthropic/claude-haiku-4.5:medium PASS 100% 240 in → 455 out (286 reasoning)
# Analysis Looking at the test results in chronological order: - ✅ 5438fce Initial commit: PASS - ✅ 4cf7fb6 add validation function: PASS - ❌ 980b62d remove validation logic: **FAIL** ← First failure - ❌ 90e5a74 add auth logging: FAIL The first commit where tests fail is **980b62d**. The commit "remove validation logic" broke the build, likely by removing necessary validation that the code depends on. **Answer: `980b62d`** (or "remove validation logic")
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 412 in → 382 out (334 reasoning)
77fa9a1
JSON Schema Structured Output
(raw) {"commit": "77fa9a1"}
anthropic/claude-haiku-4.5:none PASS 100% 211 in → 159 out (0 reasoning)
# Analysis Looking at the test results in chronological order: 1. **012eb6e** (Initial commit): PASS ✓ 2. **a0f4184** (add validation function): PASS ✓ 3. **0a21f86** (remove validation logic): FAIL ✗ 4. **ceda77d** (add auth logging): FAIL ✗ The first commit where tests started failing is **0a21f86**. ## Answer **0a21f86** (or "remove validation logic") This commit introduced the bug. The removal of validation logic broke the build, and the subsequent commit (ceda77d) did not fix the issue.
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 384 in → 13 out (0 reasoning)
27cf3c1
JSON Schema Structured Output
(raw) {"commit": "27cf3c1"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 236 in → 488 out (281 reasoning)
# Analysis Looking at the test results in chronological order: 1. **1bbbb62** (Initial commit): PASS ✓ 2. **4d2c651** (add validation function): PASS ✓ 3. **088297d** (remove validation logic): **FAIL** ✗ ← First failure 4. **eb87dfc** (add auth logging): FAIL (inherits the broken state) The build was passing through the first two commits, then broke at commit **088297d**. While the next commit also fails, that's because it built on top of the already-broken state. **Answer: `088297d`** (or "remove validation logic")
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 410 in → 382 out (318 reasoning)
2da97b0
JSON Schema Structured Output
(raw) {"commit": "2da97b0"}
anthropic/claude-opus-4.6:high PASS 100% 211 in → 71 out (0 reasoning)
The bad commit is **528d372** (`remove validation logic`). This is the first commit where tests begin failing (exit code 1), while all prior commits pass. The subsequent commit (`d8cf20a`) also fails because it builds on top of the already-broken state introduced by `528d372`.
anthropic/claude-opus-4.6:high__json_schema PASS 100% 383 in → 13 out (0 reasoning)
1af6d3e
JSON Schema Structured Output
(raw) {"commit":"1af6d3e"}
anthropic/claude-opus-4.6:low PASS 100% 209 in → 8 out (0 reasoning)
24cd6bc
anthropic/claude-opus-4.6:low__json_schema PASS 100% 385 in → 13 out (0 reasoning)
b3b1fc1
JSON Schema Structured Output
(raw) {"commit":"b3b1fc1"}
anthropic/claude-opus-4.6:max PASS 100% 205 in → 319 out (98 reasoning)
## Bad Commit Identified **`4e101db`** — *remove validation logic* ### Reasoning Examining the test results chronologically: | Commit | Subject | Result | |--------|---------|--------| | befe85e | Initial commit | ✅ PASS | | 5a90cde | add validation function | ✅ PASS | | **4e101db** | **remove validation logic** | **❌ FAIL** | | 940e58d | add auth logging | ❌ FAIL | The build was passing through `5a90cde`, then **`4e101db` is the first commit where the test fails**. This commit removed validation logic that was previously added, which broke the build. The subsequent commit (`940e58d`) also fails because it inherits the broken state.
anthropic/claude-opus-4.6:max__json_schema PASS 100% 381 in → 125 out (82 reasoning)
2cdb1ac
JSON Schema Structured Output
(raw) {"commit":"2cdb1ac"}
anthropic/claude-opus-4.6:medium PASS 100% 205 in → 8 out (0 reasoning)
08ab55f
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 377 in → 10 out (0 reasoning)
890462d
JSON Schema Structured Output
(raw) {"commit":"890462d"}
anthropic/claude-opus-4.6:none PASS 100% 213 in → 91 out (0 reasoning)
**Bad commit: `93e4ae3`** ("remove validation logic") This is the first commit where the tests change from PASS to FAIL. The previous commit (`de939f9`) passed, and `93e4ae3` introduced the failure by removing validation logic. The subsequent commit (`1f95a26`) also fails, but only because it builds on top of the already-broken state.
anthropic/claude-opus-4.6:none__json_schema PASS 100% 377 in → 10 out (0 reasoning)
2488959
JSON Schema Structured Output
(raw) {"commit":"2488959"}
anthropic/claude-opus-4.6:xhigh PASS 100% 205 in → 41 out (0 reasoning)
The bad commit is **403ac6c** (`remove validation logic`). This is the first commit where the tests start failing, meaning it introduced the bug by removing necessary validation logic.
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 387 in → 11 out (0 reasoning)
2437eba
JSON Schema Structured Output
(raw) {"commit":"2437eba"}
anthropic/claude-opus-4.7:high PASS 100% 266 in → 8 out (0 reasoning)
759777e
anthropic/claude-opus-4.7:high__json_schema PASS 100% 507 in → 16 out (0 reasoning)
4f13522
JSON Schema Structured Output
(raw) {"commit":"4f13522"}
anthropic/claude-opus-4.7:low PASS 100% 290 in → 11 out (0 reasoning)
f0f7ef1
anthropic/claude-opus-4.7:low__json_schema PASS 100% 497 in → 18 out (0 reasoning)
a7df9a0
JSON Schema Structured Output
(raw) {"commit":"a7df9a0"}
anthropic/claude-opus-4.7:max PASS 100% 280 in → 402 out (55 reasoning)
Looking at the test results, I need to find the first commit where the tests started failing: - `b030e22` Initial commit: **PASS** ✓ - `413b119` add validation function: **PASS** ✓ - `ccddaec` remove validation logic: **FAIL** ✗ ← First failure - `a6f60c8` add auth logging: **FAIL** ✗ The bad commit is **`ccddaec`** ("remove validation logic"). This commit removed validation logic that was added in the previous commit (`413b119`), which likely broke functionality that depended on that validation. The subsequent commit `a6f60c8` continues to fail because the underlying issue from `ccddaec` was never fixed.
anthropic/claude-opus-4.7:max__json_schema PASS 100% 507 in → 18 out (0 reasoning)
e0c1b00
JSON Schema Structured Output
(raw) {"commit":"e0c1b00"}
anthropic/claude-opus-4.7:medium PASS 100% 282 in → 11 out (0 reasoning)
2c1fa36
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 493 in → 16 out (0 reasoning)
abe261a
JSON Schema Structured Output
(raw) {"commit":"abe261a"}
anthropic/claude-opus-4.7:none PASS 100% 278 in → 8 out (0 reasoning)
be53e35
anthropic/claude-opus-4.7:none__json_schema PASS 100% 507 in → 17 out (0 reasoning)
6eb919b
JSON Schema Structured Output
(raw) {"commit":"6eb919b"}
anthropic/claude-opus-4.7:xhigh PASS 100% 284 in → 11 out (0 reasoning)
8e54e7b
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 501 in → 16 out (0 reasoning)
b7be247
JSON Schema Structured Output
(raw) {"commit":"b7be247"}
anthropic/claude-opus-4.8:high PASS 100% 279 in → 241 out (18 reasoning)
**Bad commit: `025084e`** ("remove validation logic") **Reasoning:** - `2c7c225` (Initial commit): PASS - `08d9ec7` (add validation function): PASS - `025084e` (remove validation logic): **FAIL** ← first failure - `5b8bb89` (add auth logging): FAIL (inherits the broken state) The build first breaks at commit `025084e`. Its parent (`08d9ec7`) passes, so this is the commit that introduced the bug. Removing the validation logic broke the build, and the subsequent commit `5b8bb89` continues to fail simply because the bug was never fixed.
anthropic/claude-opus-4.8:high__json_schema PASS 100% 486 in → 14 out (0 reasoning)
ca63556
JSON Schema Structured Output
(raw) {"commit": "ca63556"}
anthropic/claude-opus-4.8:low PASS 100% 273 in → 7 out (0 reasoning)
4d54b46
anthropic/claude-opus-4.8:low__json_schema PASS 100% 496 in → 15 out (0 reasoning)
2306e3e
JSON Schema Structured Output
(raw) {"commit":"2306e3e"}
anthropic/claude-opus-4.8:max PASS 100% 283 in → 839 out (62 reasoning)
**Bad commit: `3f9ebd2` — "remove validation logic"** ## Reasoning Looking at the test results in chronological order (oldest first), I can identify the exact transition point where the build broke: | Commit | Subject | Result | |--------|---------|--------| | `8c98f63` | Initial commit | ✅ PASS | | `f8db875` | add validation function | ✅ PASS | | **`3f9ebd2`** | **remove validation logic** | ❌ **FAIL** ← first failure | | `ff547b2` | add auth logging | ❌ FAIL | **Key observations:** 1. **The last passing commit is `f8db875`** ("add validation function"). 2. **The first failing commit is `3f9ebd2`** ("remove validation logic"). This is the transition point—the bug was introduced here. 3. **`ff547b2`** ("add auth logging") also fails, but only because it *inherits* the broken state from `3f9ebd2`. It is not the culprit itself. **Why this makes sense:** The commit immediately before (`f8db875`) *added* a validation function, and the codebase presumably depends on it. The bad commit `3f9ebd2` then *removed* that validation logic, breaking whatever code relied on it—hence the build failure. This is essentially the logic a `git bisect` would follow: find the boundary where PASS flips to FAIL. That boundary is **`3f9ebd2`**.
anthropic/claude-opus-4.8:max__json_schema PASS 100% 500 in → 107 out (61 reasoning)
93e5d17
JSON Schema Structured Output
(raw) {"commit": "93e5d17"}
anthropic/claude-opus-4.8:medium PASS 100% 281 in → 247 out (27 reasoning)
The bad commit is **3ed1c69** ("remove validation logic"). **Reasoning:** - `aaf8564` (Initial commit): PASS - `dca3d75` (add validation function): PASS - `3ed1c69` (remove validation logic): **FAIL** ← first failure - `922abf0` (add auth logging): FAIL (still broken, inherited from previous) The build transitions from PASS to FAIL at commit `3ed1c69`. Removing the validation logic broke the build, and the subsequent commit `922abf0` simply inherited the already-broken state. Therefore, `3ed1c69` is the commit that introduced the bug.
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 492 in → 15 out (0 reasoning)
c187b8b
JSON Schema Structured Output
(raw) {"commit":"c187b8b"}
anthropic/claude-opus-4.8:none PASS 100% 279 in → 131 out (0 reasoning)
Based on the test results, the build first started failing at commit `d291b7a` ("remove validation logic"). The previous commit `cbd4129` passed, and this is where the failure was introduced. **Answer: `d291b7a`** ("remove validation logic") This commit removed validation logic that the build/tests depended on, causing all subsequent commits (including `aa3dd7d`) to also fail.
anthropic/claude-opus-4.8:none__json_schema PASS 100% 498 in → 13 out (0 reasoning)
3546d03
JSON Schema Structured Output
(raw) {"commit": "3546d03"}
anthropic/claude-opus-4.8:xhigh PASS 100% 279 in → 394 out (30 reasoning)
The bad commit is: **663d689** — "remove validation logic" **Reasoning:** Looking at the test results in chronological order (oldest first): | Commit | Subject | Result | |--------|---------|--------| | 9dbf1b2 | Initial commit | ✅ PASS | | 256f2f0 | add validation function | ✅ PASS | | 663d689 | remove validation logic | ❌ FAIL | | 73efd47 | add auth logging | ❌ FAIL | The build transitions from **PASS → FAIL** at commit `663d689`. Every commit before it passes, and every commit from it onward fails. This makes `663d689` the commit that introduced the bug. The subject line ("remove validation logic") is also a strong clue — it removed logic that a subsequent step (or the tests themselves) depended on. The later commit `73efd47` still fails simply because it inherits the broken state; it isn't the root cause. **Answer: `663d689` (remove validation logic)**
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 494 in → 16 out (0 reasoning)
c02ba6d
JSON Schema Structured Output
(raw) {"commit":"c02ba6d"}
anthropic/claude-sonnet-4.6:high PASS 100% 209 in → 124 out (0 reasoning)
Looking at the test results: - `e2753be` Initial commit: **PASS** - `b334814` add validation function: **PASS** - `40e9d53` remove validation logic: **FAIL** - `ed11d16` add auth logging: **FAIL** The first failure appears at commit `40e9d53`. The subject line "remove validation logic" confirms this — it removed logic that was necessary for the build/tests to pass. **Bad commit: `40e9d53`**
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 389 in → 12 out (0 reasoning)
8e5de67
JSON Schema Structured Output
(raw) {"commit":"8e5de67"}
anthropic/claude-sonnet-4.6:low PASS 100% 209 in → 9 out (0 reasoning)
23c05b9
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 383 in → 12 out (0 reasoning)
8c76e27
JSON Schema Structured Output
(raw) {"commit":"8c76e27"}
anthropic/claude-sonnet-4.6:max PASS 100% 213 in → 350 out (108 reasoning)
## Analysis Looking at the test results chronologically: | Commit | Subject | Result | |--------|---------|--------| | 3cbd1d0 | Initial commit | ✅ PASS | | 08559b0 | add validation function | ✅ PASS | | **79b69f6** | **remove validation logic** | ❌ FAIL | | b2777cc | add auth logging | ❌ FAIL | The build was passing through `08559b0`, then failed starting at `79b69f6` and continued failing in all subsequent commits. ## Bad Commit **`79b69f6`** — *"remove validation logic"* This commit removed validation logic that the codebase depended on. The subject line itself is a strong indicator — removing logic that was previously added (and relied upon) broke the build. All commits after this one also fail, confirming this is the root cause.
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 389 in → 101 out (59 reasoning)
61faf08
JSON Schema Structured Output
(raw) {"commit":"61faf08"}
anthropic/claude-sonnet-4.6:medium PASS 100% 213 in → 10 out (0 reasoning)
4a2c3da
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 387 in → 13 out (0 reasoning)
c8e4d31
JSON Schema Structured Output
(raw) {"commit":"c8e4d31"}
anthropic/claude-sonnet-4.6:none PASS 100% 207 in → 138 out (0 reasoning)
## Analysis Looking at the test results from oldest to newest: - `893f481` Initial commit: **PASS** - `c9cc64d` add validation function: **PASS** - `7901938` remove validation logic: **FAIL** ← First failure - `35d9695` add auth logging: **FAIL** The first failing commit is `7901938`. The commit subject "remove validation logic" confirms that necessary validation logic was deleted, which broke the build. All subsequent commits also fail, indicating this is the root cause. ## Answer **`7901938`**
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 377 in → 11 out (0 reasoning)
5737c28
JSON Schema Structured Output
(raw) {"commit":"5737c28"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 215 in → 114 out (0 reasoning)
The bad commit is **a73fabe** **Reasoning:** The test results show a clear transition from PASS to FAIL at commit `a73fabe` ("remove validation logic"). The initial commit and the "add validation function" commit both pass, but starting with `a73fabe` the build breaks — and the subsequent commit `d2a2d96` continues to fail (inheriting the broken state). The subject line "remove validation logic" also directly hints that necessary validation was deleted, causing the failure.
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 375 in → 11 out (0 reasoning)
dfe65aa
JSON Schema Structured Output
(raw) {"commit":"dfe65aa"}
anthropic/claude-sonnet-5:high PASS 100% 271 in → 106 out (0 reasoning)
**62780b8** - "remove validation logic" This is the first commit where tests failed (exit code 1), immediately after the last passing commit (daac878). Since the failure persists in the next commit (032bd4f) but the transition from PASS to FAIL occurred at 62780b8, this is the commit that introduced the bug.
anthropic/claude-sonnet-5:high__json_schema PASS 100% 490 in → 14 out (0 reasoning)
28127b0
JSON Schema Structured Output
(raw) {"commit":"28127b0"}
anthropic/claude-sonnet-5:low PASS 100% 277 in → 13 out (0 reasoning)
0aae04c remove validation logic
anthropic/claude-sonnet-5:low__json_schema PASS 100% 500 in → 16 out (0 reasoning)
ca8d98c
JSON Schema Structured Output
(raw) {"commit":"ca8d98c"}
anthropic/claude-sonnet-5:medium PASS 100% 277 in → 155 out (0 reasoning)
**8aaca0c - "remove validation logic"** This is the first commit where tests fail (exit code 1), immediately following the last passing commit (a902303). Since git bisect logic points to the earliest commit where the failure appears, and the test results transition from PASS to FAIL at this commit, "remove validation logic" introduced the breaking change. The subsequent commit (b7600c2) continues to fail simply because the bug was never fixed, not because it introduced a new issue.
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 502 in → 15 out (0 reasoning)
b63daeb
JSON Schema Structured Output
(raw) {"commit":"b63daeb"}
anthropic/claude-sonnet-5:none PASS 100% 273 in → 125 out (0 reasoning)
**79112ae - "remove validation logic"** This is the first commit where tests fail (exit code 1), immediately following a passing commit (a3e710d). Since the failure persists in the next commit (731e5b9) but the transition from PASS to FAIL happens exactly at 79112ae, this is the commit that introduced the bug — likely by removing validation logic that other code depends on.
anthropic/claude-sonnet-5:none__json_schema PASS 100% 500 in → 15 out (0 reasoning)
d1a15f8
JSON Schema Structured Output
(raw) {"commit":"d1a15f8"}
anthropic/claude-sonnet-5:xhigh PASS 100% 275 in → 244 out (0 reasoning)
**79ea45b** - "remove validation logic" This commit introduced the bug. The test results show a clear transition from PASS to FAIL at this commit: - `33d8e1f` (Initial commit): PASS - `8d38925` (add validation function): PASS - `79ea45b` (remove validation logic): **FAIL** ← bug introduced here - `4d6537a` (add auth logging): FAIL (still broken, inherited from parent) Since the build was passing after "add validation function" and failed immediately after "remove validation logic," this commit is responsible for breaking the build. The subsequent commit (`4d6537a`) continues to fail simply because it was built on top of the already-broken code, not because it introduced a new issue.
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 496 in → 22 out (0 reasoning)
37a4ddf remove validation logic
JSON Schema Structured Output
(raw) {"commit": "37a4ddf remove validation logic"}
arcee-ai/trinity-large-thinking:high PASS 100% 198 in → 856 out (803 reasoning)
The bad commit is the one that first caused the test to fail, which is `77eee16` (remove validation logic). This is evident because the test passed on all commits before it and failed on this commit and all subsequent commits. **Answer:** 77eee16
arcee-ai/trinity-large-thinking:low PASS 100% 194 in → 607 out (563 reasoning)
The first commit that caused a test failure is `67b7210` with the subject "remove validation logic". This indicates that the bug was introduced when the validation logic was removed. Answer: 67b7210
arcee-ai/trinity-large-thinking:medium PASS 100% 192 in → 474 out (437 reasoning)
The bad commit is `1e245d6` (subject line: "remove validation logic"). This is the first commit where the tests fail, indicating the bug was introduced here.
arcee-ai/trinity-large-thinking:xhigh PASS 100% 194 in → 633 out (627 reasoning)
de15c5a
arcee-ai/trinity-mini:high PASS 100% 192 in → 825 out (749 reasoning)
f8f0ccb The test results show that the commit `f8f0ccb` (remove validation logic) is the first one to fail (exit code 1), while the subsequent commit `def5482` (add auth logging) also fails. Since the bug was introduced by the first failing commit, `f8f0ccb` is the bad commit.
arcee-ai/trinity-mini:high__json_schema PASS 100% 194 in → 664 out (687 reasoning)
c1bcfe0
JSON Schema Structured Output
(raw) { "commit": "c1bcfe0" }
arcee-ai/trinity-mini:low PASS 100% 192 in → 841 out (783 reasoning)
5767663
arcee-ai/trinity-mini:low__json_schema PASS 100% 198 in → 325 out (349 reasoning)
b0a3e86
JSON Schema Structured Output
(raw) { "commit": "b0a3e86" }
arcee-ai/trinity-mini:medium PASS 100% 194 in → 870 out (756 reasoning)
35b8515 The bad commit is **35b8515** ("remove validation logic"). **Reasoning:** - The initial commit (`c9bb08f`) and the commit adding validation (`7638190`) both passed tests. - The commit removing validation logic (`35b8515`) caused the build to fail (`FAIL`). - The subsequent commit adding auth logging (`9cb0dc7`) also failed, but this is a consequence of the broken state introduced by `35b8515`. - Since only one commit is identified as bad, the root cause is the removal of validation logic. **Answer:** `35b8515`
arcee-ai/trinity-mini:medium__json_schema PASS 100% 198 in → 464 out (475 reasoning)
33f283a
JSON Schema Structured Output
(raw) { "commit": "33f283a" }
arcee-ai/trinity-mini:xhigh PASS 100% 202 in → 619 out (417 reasoning)
cd729b4 **Reasoning:** 1. **Test Results Order:** The test results are listed from oldest commit to newest. The first failing commit is `cd729b4` ("remove validation logic") with exit code 1. 2. **Build Breakage:** The user states there is one bad commit that broke the build. The first commit to fail tests is `cd729b4`. 3. **Subsequent Failure:** The commit `0d44c2f` ("add auth logging") also fails, but this happens *after* the build was already broken by `cd729b4`. The failure of `0d44c2f` is likely a consequence of the existing bug introduced by `cd729b4`, not the original cause. 4. **Conclusion:** The commit that introduced the bug causing the build to break is `cd729b4`. The first 7 characters of its hash are `cd729b4`.
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 192 in → 593 out (617 reasoning)
8ac564a
JSON Schema Structured Output
(raw) { "commit": "8ac564a" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 191 in → 175 out (153 reasoning)
c3c3da2
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 270 in → 108 out (93 reasoning)
3896182
JSON Schema Structured Output
(raw) { "commit": "3896182" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 189 in → 320 out (315 reasoning)
f2656ee
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 193 in → 192 out (146 reasoning)
96a8b9d
JSON Schema Structured Output
(raw) { "commit": "96a8b9d" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 279 in → 160 out (155 reasoning)
fd032b4
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 268 in → 109 out (77 reasoning)
9e84592
JSON Schema Structured Output
(raw) { "commit": "9e84592" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 185 in → 173 out (170 reasoning)
1cdaf51
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 191 in → 238 out (242 reasoning)
730b0ae
JSON Schema Structured Output
(raw) { "commit": "730b0ae" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 193 in → 4 out (0 reasoning)
556748e
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 181 in → 9 out (0 reasoning)
4500699
JSON Schema Structured Output
(raw) {"commit": "4500699"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 185 in → 260 out (234 reasoning)
65f6416
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 185 in → 144 out (118 reasoning)
66164ac
JSON Schema Structured Output
(raw) { "commit": "66164ac" }
deepseek/deepseek-v4-flash:high PASS 100% 195 in → 113 out (106 reasoning)
baf30d8
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 195 in → 171 out (146 reasoning)
934332b
JSON Schema Structured Output
(raw) { "commit": "934332b" }
deepseek/deepseek-v4-flash:low PASS 100% 187 in → 142 out (125 reasoning)
824a8b0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 189 in → 115 out (102 reasoning)
bb2312e
JSON Schema Structured Output
(raw) {"commit": "bb2312e"}
deepseek/deepseek-v4-flash:medium PASS 100% 191 in → 153 out (146 reasoning)
e3197b6
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 191 in → 168 out (147 reasoning)
97f1d45
JSON Schema Structured Output
(raw) { "commit": "97f1d45" }
deepseek/deepseek-v4-flash:none PASS 100% 191 in → 141 out (0 reasoning)
Based on the git history and test results, the bad commit is the one that first introduced the failure. Looking at the test results chronologically (oldest first): - `0d2d4ee` - PASS - `d884cb1` - PASS - `b289d10` - FAIL (first failure) - `1fd1669` - FAIL (still failing) The commit `b289d10` is the first commit that fails the build, as indicated by "FAIL (exit code 1)" in the test results. The subsequent commit `1fd1669` continues to fail because it builds on top of the broken commit. b289d10
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 191 in → 11 out (0 reasoning)
4853718
JSON Schema Structured Output
(raw) { "commit": "4853718" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 262 in → 173 out (167 reasoning)
795ad7f
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 285 in → 9 out (0 reasoning)
0bb1621
JSON Schema Structured Output
(raw) {"commit":"0bb1621"}
deepseek/deepseek-v4-pro:high PASS 100% 191 in → 325 out (319 reasoning)
c30188e
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 189 in → 11 out (0 reasoning)
be1cc7e
JSON Schema Structured Output
(raw) {"commit": "be1cc7e"}
deepseek/deepseek-v4-pro:low PASS 100% 189 in → 64 out (0 reasoning)
Based on the test results, the first commit that failed was `6449ba6` with the subject line "remove validation logic". This commit introduced the bug that caused the build to break, and all subsequent commits (including `209a377`) continued to fail. **Bad commit:** 6449ba6
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 382 in → 248 out (237 reasoning)
94d9ae3
JSON Schema Structured Output
(raw) {"commit":"94d9ae3"}
deepseek/deepseek-v4-pro:medium PASS 100% 191 in → 193 out (160 reasoning)
8bde75d
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 193 in → 305 out (293 reasoning)
af90c7c
JSON Schema Structured Output
(raw) {"commit": "af90c7c"}
deepseek/deepseek-v4-pro:none PASS 100% 191 in → 73 out (0 reasoning)
Based on the test results, the bad commit is: **1ea382c** (or "remove validation logic") This is the first commit where the tests fail (exit code 1). The previous two commits passed, and since the failure first appears at "remove validation logic" and persists to the next commit, this is the commit that introduced the bug.
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 382 in → 9 out (0 reasoning)
108a641
JSON Schema Structured Output
(raw) {"commit": "108a641"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 268 in → 317 out (310 reasoning)
ea4f88b
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 468 in → 267 out (255 reasoning)
c0313bc
JSON Schema Structured Output
(raw) {"commit": "c0313bc"}
google/gemini-3-flash-preview:high PASS 100% 213 in → 213 out (208 reasoning)
99cbfe5
google/gemini-3-flash-preview:high__json_schema PASS 100% 218 in → 315 out (303 reasoning)
a9f6401
JSON Schema Structured Output
(raw) {"commit": "a9f6401"}
google/gemini-3-flash-preview:low PASS 100% 213 in → 132 out (0 reasoning)
The bad commit is **08f9c3a** (or **remove validation logic**). Here is the breakdown of the test results: 1. **86a9a12** (Initial commit): **PASS** 2. **76a16cc** (add validation function): **PASS** 3. **08f9c3a** (remove validation logic): **FAIL** — This is the first commit where the failure occurs. 4. **ac96524** (add auth logging): **FAIL** — The bug persists from the previous commit.
google/gemini-3-flash-preview:low__json_schema PASS 100% 208 in → 97 out (88 reasoning)
302acde
JSON Schema Structured Output
(raw) {"commit":"302acde"}
google/gemini-3-flash-preview:medium PASS 100% 211 in → 152 out (145 reasoning)
504c8e6
google/gemini-3-flash-preview:medium__json_schema PASS 100% 206 in → 235 out (227 reasoning)
7bbaff4
JSON Schema Structured Output
(raw) {"commit":"7bbaff4"}
google/gemini-3-flash-preview:xhigh PASS 100% 205 in → 212 out (208 reasoning)
ae7cea1
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 210 in → 197 out (188 reasoning)
f1bff8c
JSON Schema Structured Output
(raw) {"commit":"f1bff8c"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 215 in → 577 out (503 reasoning)
The bad commit is **5744cd0** (or "remove validation logic"). **Reasoning:** The commit `7b8253b` passed all tests, but the subsequent commit `5744cd0` caused the build to fail. This indicates that `5744cd0` introduced the breaking change.
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 212 in → 329 out (314 reasoning)
0edb778
JSON Schema Structured Output
(raw) { "commit": "0edb778" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 207 in → 209 out (130 reasoning)
The bad commit is **521c1d1**. **Reasoning:** * `0abfba0` ("add validation function") passed the tests. * `521c1d1` ("remove validation logic") caused the tests to fail. * Therefore, the break was introduced in `521c1d1`.
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 274 in → 147 out (136 reasoning)
dc067b7
JSON Schema Structured Output
(raw) {"commit": "dc067b7"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 207 in → 271 out (256 reasoning)
The bad commit is **fa3feb9** (remove validation logic).
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 268 in → 213 out (200 reasoning)
9adedbf
JSON Schema Structured Output
(raw) { "commit": "9adedbf" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 216 in → 505 out (488 reasoning)
The bad commit is **af34f16** (remove validation logic).
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 218 in → 268 out (257 reasoning)
1d092f2
JSON Schema Structured Output
(raw) {"commit":"1d092f2"}
google/gemini-3.1-pro-preview:high PASS 100% 211 in → 172 out (166 reasoning)
0ad3a49
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 278 in → 165 out (154 reasoning)
464d805
JSON Schema Structured Output
(raw) {"commit":"464d805"}
google/gemini-3.1-pro-preview:low PASS 100% 215 in → 187 out (110 reasoning)
The bad commit that introduced the bug is **a27d983** (or the subject line "**remove validation logic**"). Based on the test output, the build passes on `099887e` and `01254dd`, but begins failing with exit code 1 immediately at `a27d983`.
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 274 in → 96 out (79 reasoning)
8882a40
JSON Schema Structured Output
(raw) { "commit": "8882a40" }
google/gemini-3.1-pro-preview:medium PASS 100% 213 in → 229 out (163 reasoning)
The bad commit that introduced the bug is **2d6fd84** (or "remove validation logic"). Based on the test results, the build was passing until this commit was introduced, causing it and all subsequent commits (like `9e039d4`) to fail with exit code 1.
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 274 in → 40 out (29 reasoning)
e8edc28
JSON Schema Structured Output
(raw) {"commit": "e8edc28"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 213 in → 238 out (231 reasoning)
3e77990
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 268 in → 56 out (45 reasoning)
7ac04f4
JSON Schema Structured Output
(raw) {"commit": "7ac04f4"}
google/gemini-3.5-flash-lite:high PASS 100% 212 in → 309 out (302 reasoning)
c3c5379
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 216 in → 434 out (423 reasoning)
669f324
JSON Schema Structured Output
(raw) {"commit":"669f324"}
google/gemini-3.5-flash-lite:low PASS 100% 209 in → 103 out (98 reasoning)
abe05a7
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 278 in → 17 out (0 reasoning)
c02667c
JSON Schema Structured Output
(raw) { "commit": "c02667c" }
google/gemini-3.5-flash-lite:max PASS 100% 215 in → 382 out (375 reasoning)
362e545
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 210 in → 475 out (465 reasoning)
eea9582
JSON Schema Structured Output
(raw) {"commit": "eea9582"}
google/gemini-3.5-flash-lite:medium PASS 100% 205 in → 158 out (152 reasoning)
7c20dcb
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 274 in → 194 out (182 reasoning)
5a3f971
JSON Schema Structured Output
(raw) {"commit": "5a3f971"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 213 in → 338 out (332 reasoning)
1c5ae61
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 208 in → 602 out (593 reasoning)
06fdef5
JSON Schema Structured Output
(raw) {"commit":"06fdef5"}
google/gemini-3.5-flash:high PASS 100% 205 in → 183 out (178 reasoning)
dfdbf8a
google/gemini-3.5-flash:high__json_schema PASS 100% 268 in → 276 out (260 reasoning)
cfa1c45
JSON Schema Structured Output
(raw) { "commit": "cfa1c45" }
google/gemini-3.5-flash:low PASS 100% 203 in → 127 out (121 reasoning)
`ef5deda`
google/gemini-3.5-flash:low__json_schema PASS 100% 268 in → 95 out (83 reasoning)
111c7e9
JSON Schema Structured Output
(raw) {"commit": "111c7e9"}
google/gemini-3.5-flash:medium PASS 100% 215 in → 333 out (327 reasoning)
2ad3a87
google/gemini-3.5-flash:medium__json_schema PASS 100% 272 in → 142 out (133 reasoning)
efee54c
JSON Schema Structured Output
(raw) {"commit":"efee54c"}
google/gemini-3.5-flash:xhigh PASS 100% 211 in → 234 out (227 reasoning)
b88e656
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 270 in → 196 out (185 reasoning)
2e7f3cd
JSON Schema Structured Output
(raw) {"commit": "2e7f3cd"}
google/gemini-3.6-flash:high PASS 100% 213 in → 227 out (203 reasoning)
The bad commit that broke the build is **1732872** (or "remove validation logic").
google/gemini-3.6-flash:high__json_schema PASS 100% 274 in → 347 out (336 reasoning)
4602610
JSON Schema Structured Output
(raw) {"commit":"4602610"}
google/gemini-3.6-flash:low PASS 100% 214 in → 120 out (96 reasoning)
The bad commit that introduced the bug is **46e67a1** (or "remove validation logic").
google/gemini-3.6-flash:low__json_schema PASS 100% 274 in → 101 out (85 reasoning)
c091bda
JSON Schema Structured Output
(raw) { "commit": "c091bda" }
google/gemini-3.6-flash:max PASS 100% 212 in → 339 out (333 reasoning)
2ac78c6
google/gemini-3.6-flash:max__json_schema PASS 100% 272 in → 419 out (408 reasoning)
e03755c
JSON Schema Structured Output
(raw) {"commit":"e03755c"}
google/gemini-3.6-flash:medium PASS 100% 216 in → 263 out (239 reasoning)
The bad commit that introduced the bug is **`5b40b60`** (remove validation logic).
google/gemini-3.6-flash:medium__json_schema PASS 100% 210 in → 223 out (213 reasoning)
cff6421
JSON Schema Structured Output
(raw) {"commit":"cff6421"}
google/gemini-3.6-flash:xhigh PASS 100% 209 in → 329 out (320 reasoning)
`b911c9f`
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 216 in → 220 out (209 reasoning)
b8350b8
JSON Schema Structured Output
(raw) {"commit":"b8350b8"}
google/gemma-4-26b-a4b-it:high PASS 100% 229 in → 490 out (482 reasoning)
6a75279
google/gemma-4-26b-a4b-it:low PASS 100% 229 in → 207 out (201 reasoning)
32cafc3
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 229 in → 345 out (226 reasoning)
4650d5d
JSON Schema Structured Output
(raw) { "commit": "4650d5d" }
google/gemma-4-26b-a4b-it:medium PASS 100% 227 in → 260 out (252 reasoning)
0e4a651
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 225 in → 346 out (243 reasoning)
be655ca
JSON Schema Structured Output
(raw) { "commit": "be655ca" }
google/gemma-4-26b-a4b-it:none PASS 100% 228 in → 8 out (0 reasoning)
7e9f320
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 222 in → 16 out (0 reasoning)
bee7159
JSON Schema Structured Output
(raw) { "commit": "bee7159" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 233 in → 310 out (220 reasoning)
991867f
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 230 in → 266 out (183 reasoning)
4527098
JSON Schema Structured Output
(raw) {"commit": "4527098"}
google/gemma-4-31b-it:high PASS 100% 227 in → 194 out (1 reasoning)
346287b
google/gemma-4-31b-it:high__json_schema PASS 100% 231 in → 156 out (102 reasoning)
14b4d36
JSON Schema Structured Output
(raw) {"commit": "14b4d36"}
google/gemma-4-31b-it:low PASS 100% 226 in → 146 out (102 reasoning)
95ba8a8
google/gemma-4-31b-it:low__json_schema PASS 100% 232 in → 178 out (128 reasoning)
4421597
JSON Schema Structured Output
(raw) {"commit": "4421597"}
google/gemma-4-31b-it:medium PASS 100% 229 in → 290 out (205 reasoning)
2b71d8a
google/gemma-4-31b-it:medium__json_schema PASS 100% 230 in → 166 out (109 reasoning)
15d962b
JSON Schema Structured Output
(raw) {"commit": "15d962b"}
google/gemma-4-31b-it:none PASS 100% 224 in → 7 out (0 reasoning)
a037ca0
google/gemma-4-31b-it:none__json_schema PASS 100% 226 in → 13 out (0 reasoning)
5719327
JSON Schema Structured Output
(raw) {"commit": "5719327"}
google/gemma-4-31b-it:xhigh PASS 100% 232 in → 190 out (124 reasoning)
d91c2d4
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 228 in → 152 out (140 reasoning)
740ea4d
JSON Schema Structured Output
(raw) {"commit":"740ea4d"}
ibm-granite/granite-4.1-8b PASS 100% 202 in → 69 out
The test results indicate that the commit `04ab924` ("remove validation logic") caused the build to fail, with an exit code of 1. Since it is the first failing commit in the test results (which are listed from oldest to newest), it is the commit that introduced the bug. **Answer:** `04ab924`
ibm-granite/granite-4.1-8b__json_schema PASS 100% 202 in → 12 out
d348a34
JSON Schema Structured Output
(raw) { "commit": "d348a34" }
inclusionai/ling-2.6-flash PASS 100% 235 in → 220 out
We need to identify the first bad commit using the provided test results. Commit history from newest to oldest: 1. a786b25 add auth logging (FAIL) 2. 32633ba remove validation logic (FAIL) 3. 81b8281 add validation function (PASS) 4. 5b8b111 Initial commit (PASS) Tests run from oldest to newest: - 5b8b111: PASS - 81b8281: PASS - 32633ba: FAIL - a786b25: FAIL The first failure occurs at commit 32633ba ("remove validation logic"), which broke the build. The commit after it (a786b25) also fails, but that's a secondary failure caused by the initial breakage. The answer is the commit that first introduced the bug. 32633ba
inclusionai/ling-2.6-flash__json_schema PASS 100% 233 in → 15 out
55b956d remove validation logic
JSON Schema Structured Output
(raw) {"commit":"55b956d remove validation logic"}
liquid/lfm-2-24b-a2b PASS 100% 202 in → 36 out
The bad commit is **907d227**, as it is the only commit with a failure (`FAIL` exit code). **Answer:** `907d227`
minimax/minimax-m2.5:high PASS 100% 221 in → 282 out (297 reasoning)
3cab866
minimax/minimax-m2.5:high__json_schema PASS 100% 231 in → 464 out (452 reasoning)
b01bc14
JSON Schema Structured Output
(raw) { "commit": "b01bc14" }
minimax/minimax-m2.5:low PASS 100% 231 in → 237 out (239 reasoning)
85fd497 (remove validation logic)
minimax/minimax-m2.5:low__json_schema PASS 100% 227 in → 442 out (457 reasoning)
39563b1
JSON Schema Structured Output
(raw) { "commit": "39563b1" }
minimax/minimax-m2.5:medium PASS 100% 219 in → 538 out (552 reasoning)
994dcf9
minimax/minimax-m2.5:medium__json_schema PASS 100% 221 in → 398 out (389 reasoning)
657fef9
JSON Schema Structured Output
(raw) { "commit": "657fef9" }
minimax/minimax-m2.5:xhigh PASS 100% 223 in → 341 out (325 reasoning)
27e8a8d (remove validation logic)
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 229 in → 235 out (225 reasoning)
9400e17
JSON Schema Structured Output
(raw) { "commit": "9400e17" }
minimax/minimax-m2.7:high PASS 100% 219 in → 493 out (487 reasoning)
958b887
minimax/minimax-m2.7:high__json_schema PASS 100% 358 in → 479 out (469 reasoning)
f110a80
JSON Schema Structured Output
(raw) {"commit":"f110a80"}
minimax/minimax-m2.7:low PASS 100% 225 in → 391 out (394 reasoning)
**3b8a680** (remove validation logic) – this is the first commit that introduced the bug, causing the tests to fail.
minimax/minimax-m2.7:low__json_schema PASS 100% 360 in → 433 out (423 reasoning)
c125dff
JSON Schema Structured Output
(raw) {"commit":"c125dff"}
minimax/minimax-m2.7:medium PASS 100% 229 in → 514 out (519 reasoning)
90c0290 (remove validation logic)
minimax/minimax-m2.7:medium__json_schema PASS 100% 364 in → 379 out (369 reasoning)
98842d8
JSON Schema Structured Output
(raw) {"commit":"98842d8"}
minimax/minimax-m2.7:xhigh PASS 100% 225 in → 346 out (316 reasoning)
**52e2fa3** – “remove validation logic” introduced the bug. It’s the first commit where the tests start failing.
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 366 in → 478 out (467 reasoning)
3ed3c93
JSON Schema Structured Output
(raw) {"commit":"3ed3c93"}
minimax/minimax-m3:high PASS 100% 361 in → 202 out (145 reasoning)
The bad commit is **bed2d51** (remove validation logic). This is the first commit in the history (oldest first) where the tests went from PASS to FAIL. Removing the validation logic broke the build, and the subsequent commit (c103a36 add auth logging) also fails because it was built on top of the already-broken state.
minimax/minimax-m3:high__json_schema PASS 100% 344 in → 10 out (0 reasoning)
139bcf0
JSON Schema Structured Output
(raw) {"commit": "139bcf0"}
minimax/minimax-m3:low PASS 100% 360 in → 178 out (168 reasoning)
fba759e
minimax/minimax-m3:low__json_schema PASS 100% 350 in → 11 out (0 reasoning)
eafff5b
JSON Schema Structured Output
(raw) {"commit": "eafff5b"}
minimax/minimax-m3:medium PASS 100% 367 in → 185 out (134 reasoning)
The bad commit is **e4f75a2** ("remove validation logic"). This is the first commit where the tests transitioned from PASS to FAIL. The subsequent commit `addc32c` also fails, but that's a consequence of the build already being broken by removing the validation logic.
minimax/minimax-m3:medium__json_schema PASS 100% 346 in → 11 out (0 reasoning)
8311916
JSON Schema Structured Output
(raw) { "commit": "8311916" }
minimax/minimax-m3:xhigh PASS 100% 354 in → 220 out (146 reasoning)
**498abba** (remove validation logic) This is the first commit where the tests changed from PASS to FAIL. The prior commit (`e17ddc7` add validation function) passed, and removing the validation logic broke the build. The subsequent commit (`5001910`) remains broken only because it inherited the bug from `498abba`.
minimax/minimax-m3:xhigh__json_schema PASS 100% 348 in → 12 out (0 reasoning)
0628a69
JSON Schema Structured Output
(raw) { "commit": "0628a69" }
mistralai/devstral-2512 PASS 100% 215 in → 89 out
Based on the test results, the build was working fine until commit `105953f remove validation logic`, which introduced the failure. The subsequent commit `8bc94d7 add auth logging` also failed, but the first failure occurred at `105953f`. Therefore, the bad commit is: `105953f` (or the subject line "remove validation logic").
mistralai/devstral-2512__json_schema PASS 100% 211 in → 12 out
e5fb43d
JSON Schema Structured Output
(raw) {"commit": "e5fb43d"}
mistralai/mistral-medium-3-5:high PASS 100% 227 in → 439 out (346 reasoning)
66fe51f
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 209 in → 281 out (237 reasoning)
22f03ce
JSON Schema Structured Output
(raw) {"commit": "22f03ce"}
mistralai/mistral-medium-3-5:low PASS 100% 219 in → 492 out (446 reasoning)
da39a8a
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 207 in → 323 out (264 reasoning)
c90327e
JSON Schema Structured Output
(raw) {"commit": "c90327e"}
mistralai/mistral-medium-3-5:medium PASS 100% 219 in → 617 out (422 reasoning)
The bad commit is the first one that introduced a test failure. The test results show that `9800eba` passed, but `1839fe6` failed, meaning `1839fe6` is the commit that broke the build. The commit hash (first 7 characters) is **1839fe6**. Alternatively, the subject line is **"remove validation logic"**. 1839fe6
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 211 in → 315 out (258 reasoning)
54aa559
JSON Schema Structured Output
(raw) {"commit": "54aa559"}
mistralai/mistral-medium-3-5:none PASS 100% 221 in → 125 out (0 reasoning)
The bad commit is **f7121a2**. This is the first commit in the history (when going from oldest to newest) where the test fails. Since the commit right before it (`78a4579`) passed, but `f7121a2` failed, this means the bug was introduced by the "remove validation logic" commit. All subsequent commits (like `ec19f88`) continue to fail because they include the bad change from `f7121a2`. So the answer is: **f7121a2**
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 213 in → 13 out (0 reasoning)
c0a896c
JSON Schema Structured Output
(raw) {"commit": "c0a896c"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 223 in → 417 out (357 reasoning)
5465f86
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 209 in → 340 out (260 reasoning)
421dea1
JSON Schema Structured Output
(raw) {"commit": "421dea1"}
moonshotai/kimi-k2.6:high PASS 100% 197 in → 280 out (275 reasoning)
9b72c75
moonshotai/kimi-k2.6:high__json_schema PASS 100% 196 in → 636 out (403 reasoning)
0190d5c
JSON Schema Structured Output
(raw) { "commit": "0190d5c" }
moonshotai/kimi-k2.6:low PASS 100% 197 in → 665 out (658 reasoning)
c9a05ea
moonshotai/kimi-k2.6:low__json_schema PASS 100% 192 in → 525 out (334 reasoning)
c5f3cc4
JSON Schema Structured Output
(raw) { "commit": "c5f3cc4" }
moonshotai/kimi-k2.6:medium PASS 100% 199 in → 550 out (543 reasoning)
8f5afee
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 188 in → 597 out (204 reasoning)
4902d55
JSON Schema Structured Output
(raw) { "commit": "4902d55" }
moonshotai/kimi-k2.6:xhigh PASS 100% 193 in → 199 out (194 reasoning)
91b2d67
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 191 in → 12 out (0 reasoning)
757e966
JSON Schema Structured Output
(raw) { "commit": "757e966" }
moonshotai/kimi-k2.7-code:high PASS 100% 197 in → 155 out (147 reasoning)
5d3f46a
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 191 in → 133 out (121 reasoning)
086d540
JSON Schema Structured Output
(raw) { "commit": "086d540" }
moonshotai/kimi-k2.7-code:low PASS 100% 189 in → 132 out (122 reasoning)
4365227 (remove validation logic)
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 189 in → 132 out (129 reasoning)
6683566
JSON Schema Structured Output
(raw) { "commit": "6683566" }
moonshotai/kimi-k2.7-code:medium PASS 100% 191 in → 91 out (101 reasoning)
d35d69d
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 194 in → 136 out (67 reasoning)
022f3bb
JSON Schema Structured Output
(raw) { "commit": "022f3bb" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 191 in → 113 out (116 reasoning)
eaa03ec
moonshotai/kimi-k3:high PASS 100% 276 in → 261 out (74 reasoning)
**Bad commit: `9b3be7e` — "remove validation logic"** **Reasoning:** - `e5da634` (Initial commit): PASS - `e7c2fb0` (add validation function): PASS — last known good commit - `9b3be7e` (remove validation logic): **FAIL** — first commit where the build breaks - `af31db8` (add auth logging): FAIL — still broken, inherited from the previous commit Since the build passed at `e7c2fb0` and first failed at `9b3be7e`, the bug was introduced in `9b3be7e`. This is confirmed by the commit subject itself: it *removed* the validation logic that was added in the prior commit, which broke the build.
moonshotai/kimi-k3:high__json_schema PASS 100% 270 in → 339 out (126 reasoning)
**Bad commit: `74ebb8d` — "remove validation logic"** **Reasoning:** Working from oldest to newest, the test results show: | Commit | Subject | Result | |--------|---------|--------| | 1bda779 | Initial commit | ✅ PASS | | 5c32253 | add validation function | ✅ PASS | | **74ebb8d** | **remove validation logic** | ❌ **FAIL** ← first failure | | 4db26b2 | add auth logging | ❌ FAIL | The build passes through `5c32253` and first fails at `74ebb8d`. Since `4db26b2` (add auth logging) comes *after* the failure began, it inherited the broken state rather than causing it. The bad commit is the **first** one where the test transitions from PASS to FAIL: **`74ebb8d`**, which removed the validation logic that the build depends on.
moonshotai/kimi-k3:low PASS 100% 266 in → 343 out (162 reasoning)
**Bad commit: `ff1e7af` — "remove validation logic"** **Reasoning:** - Working backward from the test results, the transition from PASS → FAIL happens between `7eb923a` (PASS) and `ff1e7af` (FAIL). - Since `7eb923a` still passed and `ff1e7af` is the first commit that fails, the bug was introduced in `ff1e7af`. - The later commit `a250638` (add auth logging) also fails, but it only *inherited* the broken state — it didn't introduce the bug. - This is consistent with the commit subjects: the bug was introduced when the validation logic was removed, which is exactly the kind of change that would break a build depending on that validation.
moonshotai/kimi-k3:low__json_schema PASS 100% 274 in → 78 out (58 reasoning)
e17e7af
moonshotai/kimi-k3:max PASS 100% 268 in → 216 out (50 reasoning)
**Bad commit: `c877d3c` — "remove validation logic"** Reasoning: - `8548da4` (Initial commit): PASS - `455911a` (add validation function): PASS - `c877d3c` (remove validation logic): **FAIL** ← first failure - `33fbb89` (add auth logging): FAIL (inherits the bug) Since `455911a` passes and `c877d3c` is the first commit where the build fails, the bug was introduced in **`c877d3c`** ("remove validation logic"). The later commit `33fbb89` only fails because it contains the broken code from `c877d3c`.
moonshotai/kimi-k3:max__json_schema PASS 100% 264 in → 222 out (51 reasoning)
**Bad commit: `528d68f` — "remove validation logic"** **Reasoning:** - `dd19741` (Initial commit): PASS - `0628133` (add validation function): PASS - `528d68f` (remove validation logic): **FAIL** ← first failing commit - `d714ac1` (add auth logging): FAIL (still broken, but it inherited the failure) Since the build passed at `0628133` and first failed at `528d68f`, commit `528d68f` is the one that introduced the bug — fittingly, it removed the validation logic that the tests depend on. The later commit `d714ac1` only added logging and didn't cause the failure.
moonshotai/kimi-k3:medium PASS 100% 274 in → 415 out (147 reasoning)
**Bad commit: `2056f51` — "remove validation logic"** **Reasoning:** Walking through the history chronologically (oldest first): | Commit | Subject | Result | |--------|---------|--------| | d8259e0 | Initial commit | ✅ PASS | | 6167e5f | add validation function | ✅ PASS | | **2056f51** | **remove validation logic** | ❌ **FAIL** | | 5a0b333 | add auth logging | ❌ FAIL | The tests pass at `6167e5f` and first fail at `2056f51`, so `2056f51` is the commit that introduced the bug. The newest commit `5a0b333` ("add auth logging") also fails, but that's because it *inherited* the broken state — it merely added logging on top of the already-broken code. The subject line also corroborates this: the validation function was added in `6167e5f` (tests pass), then the validation logic was *removed* in `2056f51`, which immediately breaks the build. A bisect would converge on `2056f51` as the first bad commit.
moonshotai/kimi-k3:medium__json_schema PASS 100% 270 in → 289 out (94 reasoning)
**Bad commit: `4b357c2` — "remove validation logic"** **Reasoning:** - `4836ec0` (Initial commit) → PASS - `c1a3041` (add validation function) → PASS - `4b357c2` (remove validation logic) → **FAIL** ← first failure - `664a408` (add auth logging) → FAIL (still broken, but not the cause) The build transitions from PASS to FAIL at commit `4b357c2`. The subsequent commit `664a408` also fails, but that's because it inherits the broken state — it didn't introduce the bug. The commit subject also hints at the problem: removing validation logic likely broke functionality that tests depend on. To fix: `git revert 4b357c2` or restore the validation logic removed in that commit.
moonshotai/kimi-k3:xhigh PASS 100% 266 in → 170 out (80 reasoning)
**7b00989** (remove validation logic) The first failing commit is `7b00989`. The previous commits (`dea0b20` and `cfa3d61`) both pass, so the bug was introduced when the validation logic was removed. The subsequent commit `4ade623` also fails, but it inherited the broken state rather than causing it.
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 374 in → 107 out (83 reasoning)
0a6976b
JSON Schema Structured Output
(raw) {"commit":"0a6976b"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 228 in → 317 out (304 reasoning)
**b173a8b** – the commit that removed validation logic introduced the failure.
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 226 in → 405 out (347 reasoning)
d22407e
JSON Schema Structured Output
(raw) {"commit": "d22407e" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 230 in → 426 out (361 reasoning)
`9b2a36e`
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 226 in → 166 out (167 reasoning)
e1156f9
JSON Schema Structured Output
(raw) { "commit": "e1156f9" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 226 in → 369 out (351 reasoning)
**13863cb** (commit subject: *“remove validation logic”*)
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 216 in → 313 out (310 reasoning)
5cb7cab
JSON Schema Structured Output
(raw) { "commit": "5cb7cab" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 222 in → 10 out (0 reasoning)
16a26ac remove validation logic
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 220 in → 20 out (0 reasoning)
e53257f
JSON Schema Structured Output
(raw) { "commit": "e53257f" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 226 in → 210 out (174 reasoning)
848688c
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 222 in → 195 out (226 reasoning)
54a261f
JSON Schema Structured Output
(raw) { "commit": "54a261f" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 226 in → 154 out (126 reasoning)
a7657a9
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 230 in → 292 out (225 reasoning)
891b185
JSON Schema Structured Output
(raw) { "commit": "891b185" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 226 in → 221 out (180 reasoning)
2dc905f
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 228 in → 185 out (134 reasoning)
086756a
JSON Schema Structured Output
(raw) { "commit": "086756a" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 222 in → 235 out (199 reasoning)
ac60add
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 226 in → 191 out (161 reasoning)
5cfc24f
JSON Schema Structured Output
(raw) { "commit": "5cfc24f" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 226 in → 214 out (0 reasoning)
Based on the test results and commit history (oldest to newest): - **7243370 Initial commit**: PASS - **d03cd12 add validation function**: PASS - **0b78c32 remove validation logic**: FAIL - **dd85941 add auth logging**: FAIL The first failing commit is **0b78c32 remove validation logic**. Since the prior commit (`d03cd12`) passed, and this one failed, the bug was introduced in this commit. The test continues to fail in the next commit (`dd85941`), which is expected because it builds on the broken state. ### Answer: **0b78c32** (first 7 characters of the bad commit hash) or equivalently: **remove validation logic** (the commit subject line) Both are acceptable per the instructions. **Final Answer: 0b78c32**
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 224 in → 15 out (0 reasoning)
dc87502
JSON Schema Structured Output
(raw) { "commit": "dc87502" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 220 in → 213 out (200 reasoning)
fba7b4c
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 220 in → 160 out (131 reasoning)
c68adf2
JSON Schema Structured Output
(raw) {"commit": "c68adf2"}
openai/gpt-5.4-mini:high PASS 100% 195 in → 59 out (47 reasoning)
e6d04b5
openai/gpt-5.4-mini:high__json_schema PASS 100% 230 in → 82 out (63 reasoning)
aa630e3
JSON Schema Structured Output
(raw) {"commit":"aa630e3"}
openai/gpt-5.4-mini:low PASS 100% 189 in → 34 out (23 reasoning)
f3460a0
openai/gpt-5.4-mini:low__json_schema PASS 100% 234 in → 66 out (46 reasoning)
b5798d5
JSON Schema Structured Output
(raw) {"commit":"b5798d5"}
openai/gpt-5.4-mini:medium PASS 100% 189 in → 43 out (33 reasoning)
6454b14
openai/gpt-5.4-mini:medium__json_schema PASS 100% 234 in → 79 out (59 reasoning)
7db3f57
JSON Schema Structured Output
(raw) {"commit":"7db3f57"}
openai/gpt-5.4-mini:none PASS 100% 189 in → 8 out (0 reasoning)
01f638d
openai/gpt-5.4-mini:none__json_schema PASS 100% 230 in → 18 out (0 reasoning)
f23dc2f
JSON Schema Structured Output
(raw) {"commit":"f23dc2f"}
openai/gpt-5.4-mini:xhigh PASS 100% 191 in → 117 out (106 reasoning)
88a4ee1
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 234 in → 129 out (109 reasoning)
6f2373b
JSON Schema Structured Output
(raw) {"commit":"6f2373b"}
openai/gpt-5.4-nano:high PASS 100% 197 in → 69 out (59 reasoning)
66e9815
openai/gpt-5.4-nano:high__json_schema PASS 100% 226 in → 68 out (50 reasoning)
7cab590
JSON Schema Structured Output
(raw) {"commit":"7cab590"}
openai/gpt-5.4-nano:low PASS 100% 191 in → 35 out (26 reasoning)
867498e
openai/gpt-5.4-nano:low__json_schema PASS 100% 232 in → 18 out (0 reasoning)
e9a451f
JSON Schema Structured Output
(raw) {"commit":"e9a451f"}
openai/gpt-5.4-nano:medium PASS 100% 187 in → 36 out (24 reasoning)
cff4318 remove validation logic
openai/gpt-5.4-nano:medium__json_schema PASS 100% 228 in → 50 out (31 reasoning)
eb2075c
JSON Schema Structured Output
(raw) {"commit":"eb2075c"}
openai/gpt-5.4-nano:none PASS 100% 191 in → 8 out (0 reasoning)
6e541ca
openai/gpt-5.4-nano:none__json_schema PASS 100% 230 in → 17 out (0 reasoning)
8278b82
JSON Schema Structured Output
(raw) {"commit":"8278b82"}
openai/gpt-5.4-nano:xhigh PASS 100% 187 in → 124 out (114 reasoning)
aa83ee4
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 230 in → 167 out (149 reasoning)
bef441e
JSON Schema Structured Output
(raw) {"commit":"bef441e"}
openai/gpt-5.4:high PASS 100% 191 in → 73 out (62 reasoning)
c67c76c
openai/gpt-5.4:high__json_schema PASS 100% 238 in → 83 out (62 reasoning)
e32a0d8
JSON Schema Structured Output
(raw) {"commit":"e32a0d8"}
openai/gpt-5.4:low PASS 100% 195 in → 33 out (22 reasoning)
505c3a2
openai/gpt-5.4:low__json_schema PASS 100% 232 in → 43 out (24 reasoning)
9cb7203
JSON Schema Structured Output
(raw) {"commit":"9cb7203"}
openai/gpt-5.4:medium PASS 100% 193 in → 53 out (43 reasoning)
398ac8a
openai/gpt-5.4:medium__json_schema PASS 100% 228 in → 39 out (20 reasoning)
2c64324
JSON Schema Structured Output
(raw) {"commit":"2c64324"}
openai/gpt-5.4:none PASS 100% 195 in → 9 out (0 reasoning)
d3eb4fd
openai/gpt-5.4:none__json_schema PASS 100% 234 in → 18 out (0 reasoning)
02c4df2
JSON Schema Structured Output
(raw) {"commit":"02c4df2"}
openai/gpt-5.4:xhigh PASS 100% 187 in → 62 out (53 reasoning)
a273261
openai/gpt-5.4:xhigh__json_schema PASS 100% 230 in → 150 out (131 reasoning)
b02d331
JSON Schema Structured Output
(raw) {"commit":"b02d331"}
openai/gpt-5.5:high PASS 100% 185 in → 39 out (30 reasoning)
288987b
openai/gpt-5.5:high__json_schema PASS 100% 234 in → 118 out (96 reasoning)
e5d8e7d
JSON Schema Structured Output
(raw) {"commit":"e5d8e7d"}
openai/gpt-5.5:low PASS 100% 187 in → 8 out (0 reasoning)
7b322be
openai/gpt-5.5:low__json_schema PASS 100% 230 in → 16 out (0 reasoning)
324befd
JSON Schema Structured Output
(raw) {"commit":"324befd"}
openai/gpt-5.5:medium PASS 100% 187 in → 33 out (24 reasoning)
695687b
openai/gpt-5.5:medium__json_schema PASS 100% 238 in → 18 out (0 reasoning)
0e26adf
JSON Schema Structured Output
(raw) {"commit":"0e26adf"}
openai/gpt-5.5:none PASS 100% 193 in → 9 out (0 reasoning)
7acec3a
openai/gpt-5.5:none__json_schema PASS 100% 234 in → 16 out (0 reasoning)
951513b
JSON Schema Structured Output
(raw) {"commit":"951513b"}
openai/gpt-5.5:xhigh PASS 100% 187 in → 72 out (62 reasoning)
8ba674b
openai/gpt-5.5:xhigh__json_schema PASS 100% 236 in → 56 out (37 reasoning)
3db7013
JSON Schema Structured Output
(raw) {"commit":"3db7013"}
openai/gpt-5.6-luna:high PASS 100% 191 in → 9 out (0 reasoning)
da8f6bd
openai/gpt-5.6-luna:high__json_schema PASS 100% 232 in → 17 out (0 reasoning)
6775b81
JSON Schema Structured Output
(raw) {"commit":"6775b81"}
openai/gpt-5.6-luna:low PASS 100% 189 in → 7 out (0 reasoning)
257303f
openai/gpt-5.6-luna:low__json_schema PASS 100% 234 in → 17 out (0 reasoning)
f70696e
JSON Schema Structured Output
(raw) {"commit":"f70696e"}
openai/gpt-5.6-luna:max PASS 100% 199 in → 71 out (54 reasoning)
b2c3b98 (remove validation logic)
openai/gpt-5.6-luna:max__json_schema PASS 100% 236 in → 17 out (0 reasoning)
7c35681
JSON Schema Structured Output
(raw) {"commit":"7c35681"}
openai/gpt-5.6-luna:medium PASS 100% 191 in → 9 out (0 reasoning)
6a414d3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 232 in → 18 out (0 reasoning)
5b393e0
JSON Schema Structured Output
(raw) {"commit":"5b393e0"}
openai/gpt-5.6-luna:none PASS 100% 193 in → 8 out (0 reasoning)
1b33608
openai/gpt-5.6-luna:none__json_schema PASS 100% 244 in → 19 out (0 reasoning)
a4c3e81
JSON Schema Structured Output
(raw) {"commit":"a4c3e81"}
openai/gpt-5.6-luna:xhigh PASS 100% 185 in → 9 out (0 reasoning)
2a9d248
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 230 in → 16 out (0 reasoning)
ae488ea
JSON Schema Structured Output
(raw) {"commit":"ae488ea"}
openai/gpt-5.6-sol:high PASS 100% 197 in → 10 out (0 reasoning)
9e1dce7
openai/gpt-5.6-sol:high__json_schema PASS 100% 230 in → 17 out (0 reasoning)
20dd6ba
JSON Schema Structured Output
(raw) {"commit":"20dd6ba"}
openai/gpt-5.6-sol:low PASS 100% 191 in → 8 out (0 reasoning)
4df6065
openai/gpt-5.6-sol:low__json_schema PASS 100% 232 in → 18 out (0 reasoning)
f36dd9d
JSON Schema Structured Output
(raw) {"commit":"f36dd9d"}
openai/gpt-5.6-sol:max PASS 100% 193 in → 9 out (0 reasoning)
4f1705b
openai/gpt-5.6-sol:max__json_schema PASS 100% 232 in → 18 out (0 reasoning)
bf4d61e
JSON Schema Structured Output
(raw) {"commit":"bf4d61e"}
openai/gpt-5.6-sol:medium PASS 100% 195 in → 8 out (0 reasoning)
1cdef00
openai/gpt-5.6-sol:medium__json_schema PASS 100% 232 in → 17 out (0 reasoning)
21a76bc
JSON Schema Structured Output
(raw) {"commit":"21a76bc"}
openai/gpt-5.6-sol:none PASS 100% 193 in → 8 out (0 reasoning)
1af0adc
openai/gpt-5.6-sol:none__json_schema PASS 100% 236 in → 19 out (0 reasoning)
a5e7a47
JSON Schema Structured Output
(raw) {"commit":"a5e7a47"}
openai/gpt-5.6-sol:xhigh PASS 100% 189 in → 8 out (0 reasoning)
41fead0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 240 in → 18 out (0 reasoning)
ca4d18a
JSON Schema Structured Output
(raw) {"commit":"ca4d18a"}
openai/gpt-5.6-terra:high PASS 100% 187 in → 10 out (0 reasoning)
5011934 remove validation logic
openai/gpt-5.6-terra:high__json_schema PASS 100% 230 in → 18 out (0 reasoning)
d71e0bf
JSON Schema Structured Output
(raw) {"commit":"d71e0bf"}
openai/gpt-5.6-terra:low PASS 100% 187 in → 7 out (0 reasoning)
e046710
openai/gpt-5.6-terra:low__json_schema PASS 100% 232 in → 17 out (0 reasoning)
b509a64
JSON Schema Structured Output
(raw) {"commit":"b509a64"}
openai/gpt-5.6-terra:max PASS 100% 191 in → 45 out (35 reasoning)
b9ef727
openai/gpt-5.6-terra:max__json_schema PASS 100% 232 in → 18 out (0 reasoning)
a3955f7
JSON Schema Structured Output
(raw) {"commit":"a3955f7"}
openai/gpt-5.6-terra:medium PASS 100% 193 in → 7 out (0 reasoning)
02978af
openai/gpt-5.6-terra:medium__json_schema PASS 100% 228 in → 16 out (0 reasoning)
181a502
JSON Schema Structured Output
(raw) {"commit":"181a502"}
openai/gpt-5.6-terra:none PASS 100% 193 in → 10 out (0 reasoning)
1df4f5e
openai/gpt-5.6-terra:none__json_schema PASS 100% 234 in → 18 out (0 reasoning)
7e1b983
JSON Schema Structured Output
(raw) {"commit":"7e1b983"}
openai/gpt-5.6-terra:xhigh PASS 100% 183 in → 7 out (0 reasoning)
c283169
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 228 in → 17 out (0 reasoning)
6958e46
JSON Schema Structured Output
(raw) {"commit":"6958e46"}
openai/gpt-oss-120b:high PASS 100% 248 in → 489 out (471 reasoning)
c077e83 – remove validation logic
openai/gpt-oss-120b:high__json_schema PASS 100% 250 in → 344 out (365 reasoning)
6b6f397
JSON Schema Structured Output
(raw) { "commit": "6b6f397" }
openai/gpt-oss-120b:low PASS 100% 250 in → 63 out (44 reasoning)
db34d30 (remove validation logic)
openai/gpt-oss-120b:low__json_schema PASS 100% 245 in → 64 out (37 reasoning)
7739fc9
JSON Schema Structured Output
(raw) { "commit": "7739fc9" }
openai/gpt-oss-120b:medium PASS 100% 244 in → 200 out (175 reasoning)
**Bad commit:** `9116f48` — *remove validation logic*
openai/gpt-oss-120b:medium__json_schema PASS 100% 241 in → 158 out (145 reasoning)
8a6de12
JSON Schema Structured Output
(raw) { "commit": "8a6de12" }
openai/gpt-oss-120b:xhigh PASS 100% 248 in → 665 out (686 reasoning)
02cfc07
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 239 in → 240 out (245 reasoning)
21e6af5
JSON Schema Structured Output
(raw) { "commit": "21e6af5" }
openai/gpt-oss-20b:high PASS 100% 252 in → 706 out (684 reasoning)
The failing commit that introduced the bug is **7fd3032** – the “remove validation logic” commit.
openai/gpt-oss-20b:high__json_schema PASS 100% 250 in → 372 out (377 reasoning)
774bb87
JSON Schema Structured Output
(raw) { "commit":"774bb87" }
openai/gpt-oss-20b:low PASS 100% 250 in → 107 out (94 reasoning)
04380dc
openai/gpt-oss-20b:low__json_schema PASS 100% 248 in → 89 out (63 reasoning)
de26f18
JSON Schema Structured Output
(raw) { "commit": "de26f18" }
openai/gpt-oss-20b:medium PASS 100% 252 in → 291 out (317 reasoning)
dc3dcb8
openai/gpt-oss-20b:medium__json_schema PASS 100% 237 in → 159 out (136 reasoning)
670964b
JSON Schema Structured Output
(raw) { "commit": "670964b" }
openai/gpt-oss-20b:xhigh PASS 100% 258 in → 715 out (706 reasoning)
b91f065 (remove validation logic)
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 235 in → 357 out (362 reasoning)
cf05d94
JSON Schema Structured Output
(raw) {"commit":"cf05d94"}
poolside/laguna-m.1:high PASS 100% 226 in → 354 out (286 reasoning)
The bad commit is **766b1d3**. This commit removes validation logic, which directly precedes the first test failure in the sequence. The subsequent commit (54f5142) also fails, but the root cause originates from the removal of validation in 766b1d3.
poolside/laguna-m.1:high__json_schema PASS 100% 226 in → 339 out (327 reasoning)
9688ace
JSON Schema Structured Output
(raw) {"commit": "9688ace"}
poolside/laguna-m.1:low PASS 100% 224 in → 597 out (587 reasoning)
c1012c5
poolside/laguna-m.1:low__json_schema PASS 100% 226 in → 591 out (578 reasoning)
61b7b98
JSON Schema Structured Output
(raw) {"commit":"61b7b98"}
poolside/laguna-m.1:medium PASS 100% 224 in → 409 out (400 reasoning)
0aa68a5
poolside/laguna-m.1:medium__json_schema PASS 100% 230 in → 538 out (524 reasoning)
91b1d72
JSON Schema Structured Output
(raw) {"commit": "91b1d72"}
poolside/laguna-m.1:none PASS 100% 224 in → 184 out (0 reasoning)
Based on the git log and test results, I can see that: 1. The initial commit (ae49dc7) passed all tests 2. The commit "add validation function" (cb5957a) also passed 3. The commit "remove validation logic" (1d177d6) failed the tests 4. The most recent commit "add auth logging" (2072967) also failed Since both commits after cb5957a are failing, and the commit "remove validation logic" (1d177d6) directly follows the passing commit "add validation function" (cb5957a), the bad commit that introduced the bug is most likely the one that removed the validation logic. **Answer: 1d177d6** (or "remove validation logic")
poolside/laguna-m.1:none__json_schema PASS 100% 218 in → 12 out (0 reasoning)
8169ac2
JSON Schema Structured Output
(raw) {"commit": "8169ac2"}
poolside/laguna-m.1:xhigh PASS 100% 226 in → 356 out (346 reasoning)
10183f5
poolside/laguna-m.1:xhigh__json_schema PASS 100% 212 in → 506 out (496 reasoning)
1bacfef
JSON Schema Structured Output
(raw) {"commit": "1bacfef"}
poolside/laguna-xs-2.1:high PASS 100% 228 in → 446 out (438 reasoning)
c46392a
poolside/laguna-xs-2.1:high__json_schema PASS 100% 222 in → 544 out (531 reasoning)
6ec71d0
JSON Schema Structured Output
(raw) {"commit": "6ec71d0"}
poolside/laguna-xs-2.1:low PASS 100% 222 in → 365 out (255 reasoning)
The bad commit is identified by analyzing the test results. The initial commit (25bcec8) and the commit adding validation (1e3a7c1) both pass. The next commit (3d32bfe) removes validation logic and causes the test to fail. Since this is the first failure in the sequence, it indicates that this commit introduced the bug. The subsequent commit (6c2b529) also fails but is not the root cause. **Answer:** 3d32bfe
poolside/laguna-xs-2.1:low__json_schema PASS 100% 226 in → 267 out (255 reasoning)
cf40ce9
JSON Schema Structured Output
(raw) {"commit": "cf40ce9"}
poolside/laguna-xs-2.1:medium PASS 100% 218 in → 760 out (625 reasoning)
The bad commit is **c90f6ce** ("remove validation logic"). **Reasoning:** 1. The initial commit and `11be7b5` ("add validation function") both passed tests. 2. `c90f6ce` ("remove validation logic") is the first commit to cause a test failure (exit code 1). 3. Subsequent commits (e.g., `23ea7fa`) inherited the broken state but were not the original cause. The first failing commit after passing tests is the root cause. **Answer:** c90f6ce
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 228 in → 381 out (368 reasoning)
d991cd6
JSON Schema Structured Output
(raw) {"commit": "d991cd6"}
poolside/laguna-xs-2.1:none PASS 100% 226 in → 7 out (0 reasoning)
2871b72
poolside/laguna-xs-2.1:none__json_schema PASS 100% 228 in → 13 out (0 reasoning)
b469e85
JSON Schema Structured Output
(raw) {"commit": "b469e85"}
poolside/laguna-xs-2.1:xhigh PASS 100% 226 in → 532 out (374 reasoning)
The bad commit is **6a6e8b9**. Here's the breakdown: - The test passed after `d27c818` (Initial commit) and `63692ce` (add validation function). - The test **first failed** at `6a6e8b9` (remove validation logic), indicating this commit introduced the breaking change. - Subsequent commits like `0b962ee` (add auth logging) also failed, but they are built on top of the broken state caused by `6a6e8b9`. The root cause is the removal of validation logic, which likely broke the build. **Answer:** 6a6e8b9
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 220 in → 395 out (379 reasoning)
fefa8b2
JSON Schema Structured Output
(raw) { "commit": "fefa8b2" }
poolside/laguna-xs.2:high PASS 100% 263 in → 389 out (233 reasoning)
Looking at the test results chronologically: 1. **583fcc4 Initial commit**: PASS 2. **4b38e52 add validation function**: PASS 3. **c2f36d6 remove validation logic**: FAIL ← Tests start failing here 4. **214c7e1 add auth logging**: FAIL The build was passing after commit `4b38e52`, but failed after `c2f36d6`. This indicates that commit `c2f36d6` (subject: "remove validation logic") introduced the bug by removing validation logic that was needed for the build to pass. **Bad commit: c2f36d6**
poolside/laguna-xs.2:high__json_schema PASS 100% 259 in → 329 out (314 reasoning)
dcf9dae
JSON Schema Structured Output
(raw) { "commit": "dcf9dae" }
poolside/laguna-xs.2:low PASS 100% 251 in → 321 out (207 reasoning)
Looking at the test results chronologically: 1. `5336ddf Initial commit`: PASS 2. `105cbaf add validation function`: PASS 3. `ffd08e0 remove validation logic`: FAIL ← **First failure** 4. `bad1f6f add auth logging`: FAIL The bug was introduced by commit `ffd08e0` titled "remove validation logic" - this is the first commit where tests started failing. **Answer: ffd08e0**
poolside/laguna-xs.2:low__json_schema PASS 100% 265 in → 277 out (263 reasoning)
c3f6d36
JSON Schema Structured Output
(raw) {"commit": "c3f6d36"}
poolside/laguna-xs.2:medium PASS 100% 261 in → 393 out (328 reasoning)
Based on the test results, I can see that the build was passing up until commit `8f84abc`, but then failed starting with commit `979bc16`. The commit that introduced the bug is: **979bc16** (remove validation logic)
poolside/laguna-xs.2:medium__json_schema PASS 100% 261 in → 238 out (226 reasoning)
cbad9e4
JSON Schema Structured Output
(raw) {"commit": "cbad9e4"}
poolside/laguna-xs.2:none PASS 100% 259 in → 178 out (0 reasoning)
Looking at the test results, I can see that: - Commit `7d40527` (Initial commit): PASS - Commit `4ccd4ff` (add validation function): PASS - Commit `6b9929e` (remove validation logic): FAIL - Commit `52a5ab7` (add auth logging): FAIL The build passed up to commit `4ccd4ff` but failed starting with commit `6b9929e`. This commit message "remove validation logic" suggests it removed important validation that was causing the build to fail. Even though commit `52a5ab7` also fails, it was built on top of the already broken state from `6b9929e`. The bad commit is: **6b9929e**
poolside/laguna-xs.2:none__json_schema PASS 100% 263 in → 13 out (0 reasoning)
4853718
JSON Schema Structured Output
(raw) {"commit": "4853718"}
poolside/laguna-xs.2:xhigh PASS 100% 263 in → 378 out (205 reasoning)
Based on the test results, I can see that: 1. Commit `799b1aa` (Initial commit): **PASS** 2. Commit `4d2e397` (add validation function): **PASS** 3. Commit `5e5c724` (remove validation logic): **FAIL** 4. Commit `0ad51a8` (add auth logging): **FAIL** The tests were passing up until commit `4d2e397`, but then failed starting with commit `5e5c724`. This indicates that commit `5e5c724` ("remove validation logic") introduced the bug by removing the validation logic that was working correctly before. **Bad commit: 5e5c724**
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 261 in → 332 out (319 reasoning)
04df12b
JSON Schema Structured Output
(raw) {"commit": "04df12b"}
qwen/qwen3.6-27b:high PASS 100% 224 in → 652 out (523 reasoning)
9840d10
qwen/qwen3.6-27b:high__json_schema PASS 100% 220 in → 792 out (634 reasoning)
4335783
JSON Schema Structured Output
(raw) { "commit": "4335783" }
qwen/qwen3.6-27b:low PASS 100% 218 in → 664 out (546 reasoning)
f490cf8
qwen/qwen3.6-27b:low__json_schema PASS 100% 214 in → 570 out (1 reasoning)
bd825a9
JSON Schema Structured Output
(raw) { "commit": "bd825a9" }
qwen/qwen3.6-27b:medium PASS 100% 222 in → 827 out (672 reasoning)
662e0bf
qwen/qwen3.6-27b:none PASS 100% 216 in → 7 out (0 reasoning)
36069cd
qwen/qwen3.6-27b:none__json_schema PASS 100% 228 in → 18 out (0 reasoning)
1e5b792
JSON Schema Structured Output
(raw) { "commit": "1e5b792" }
qwen/qwen3.6-27b:xhigh PASS 100% 224 in → 626 out (458 reasoning)
57328c5
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 222 in → 14 out (0 reasoning)
bcc4ca9
JSON Schema Structured Output
(raw) { "commit": "bcc4ca9" }
qwen/qwen3.6-35b-a3b:high PASS 100% 220 in → 372 out (361 reasoning)
5cd6807
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 218 in → 811 out (672 reasoning)
9103b25
JSON Schema Structured Output
(raw) { "commit": "9103b25" }
qwen/qwen3.6-35b-a3b:low PASS 100% 216 in → 647 out (485 reasoning)
16de611
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 224 in → 534 out (445 reasoning)
28b832d
JSON Schema Structured Output
(raw) { "commit": "28b832d" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 216 in → 333 out (278 reasoning)
be06ffb
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 218 in → 638 out (510 reasoning)
9d90f47
JSON Schema Structured Output
(raw) { "commit": "9d90f47" }
qwen/qwen3.6-35b-a3b:none PASS 100% 226 in → 7 out (0 reasoning)
7b9f749
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 224 in → 17 out (0 reasoning)
a909ed2
JSON Schema Structured Output
(raw) { "commit": "a909ed2" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 214 in → 367 out (333 reasoning)
a99ceb1
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 220 in → 408 out (328 reasoning)
d93bfe1
JSON Schema Structured Output
(raw) { "commit": "d93bfe1" }
qwen/qwen3.6-flash:high PASS 100% 216 in → 1,451 out (1,436 reasoning)
4ccab0e (remove validation logic)
qwen/qwen3.6-flash:low PASS 100% 214 in → 294 out (283 reasoning)
0e22ed8
qwen/qwen3.6-flash:medium PASS 100% 224 in → 477 out (465 reasoning)
88038c6
qwen/qwen3.6-flash:none PASS 100% 224 in → 6 out (0 reasoning)
5083ed3
qwen/qwen3.6-flash:xhigh PASS 100% 222 in → 383 out (371 reasoning)
c5402f6
qwen/qwen3.7-flash:high PASS 100% 222 in → 376 out (364 reasoning)
7060b47
qwen/qwen3.7-flash:low PASS 100% 220 in → 458 out (447 reasoning)
85b45fb
qwen/qwen3.7-flash:max PASS 100% 220 in → 650 out (638 reasoning)
e275623
qwen/qwen3.7-flash:medium PASS 100% 216 in → 389 out (379 reasoning)
e7cf5cb
qwen/qwen3.7-flash:none PASS 100% 226 in → 6 out (0 reasoning)
564e9ca
qwen/qwen3.7-flash:xhigh PASS 100% 218 in → 476 out (465 reasoning)
90dcf6d
qwen/qwen3.7-max:high PASS 100% 220 in → 150 out (139 reasoning)
9ab0c31
qwen/qwen3.7-max:low PASS 100% 218 in → 301 out (174 reasoning)
The bad commit is **cc66250** ("remove validation logic"). **Reasoning:** Looking at the test results chronologically, the build was passing through commit `264abc7` (add validation function) and first failed at commit `cc66250` (remove validation logic). This is where the transition from PASS → FAIL occurs, indicating that removing the validation logic is what broke the build. The subsequent commit `0577e0a` (add auth logging) continues to fail because it's built on top of the already-broken commit.
qwen/qwen3.7-max:medium PASS 100% 214 in → 166 out (156 reasoning)
f1cac5e
qwen/qwen3.7-max:none PASS 100% 226 in → 7 out (0 reasoning)
7120e0f
qwen/qwen3.7-max:xhigh PASS 100% 222 in → 296 out (181 reasoning)
The bad commit is **776cb08** ("remove validation logic"). The test results show that the build was passing through commit `7839f6f` (add validation function) and first failed at commit `776cb08` (remove validation logic). This is where the regression was introduced — likely by removing validation that the build/tests depended on. The subsequent commit `966cdaa` continues to fail because the bug from `776cb08` was never fixed.
qwen/qwen3.7-plus:high PASS 100% 224 in → 229 out (217 reasoning)
05f9f7f
qwen/qwen3.7-plus:low PASS 100% 222 in → 391 out (218 reasoning)
The bad commit is **7890573** (`remove validation logic`). **Reasoning:** Looking at the test results chronologically: 1. `1b7f144` Initial commit → **PASS** ✅ 2. `3726d52` add validation function → **PASS** ✅ 3. `7890573` remove validation logic → **FAIL** ❌ 4. `f1cce82` add auth logging → **FAIL** ❌ The first transition from PASS to FAIL occurs at commit `7890573`. This is where the validation logic was removed, which broke the build. All subsequent commits (including `f1cce82`) also fail because they build on top of this broken state.
qwen/qwen3.7-plus:medium PASS 100% 222 in → 372 out (233 reasoning)
The bad commit is **6876502** ("remove validation logic"). **Reasoning:** - `b04bd7e` → PASS ✅ - `98c5265` → PASS ✅ - `6876502` → FAIL ❌ ← bug introduced here - `7e07bc7` → FAIL ❌ (inherits the bug) The first failing commit is `6876502`, where the validation logic was removed, which broke the build. The previous commit `98c5265` was the last known good commit.
qwen/qwen3.7-plus:none PASS 100% 222 in → 6 out (0 reasoning)
3ca0d8b
qwen/qwen3.7-plus:xhigh PASS 100% 220 in → 426 out (207 reasoning)
## Identifying the Bad Commit Looking at the test results chronologically (oldest to newest): | Commit | Subject | Test Result | |--------|---------|-------------| | `4c2c1c6` | Initial commit | ✅ PASS | | `f841543` | add validation function | ✅ PASS | | `bf1e41e` | remove validation logic | ❌ FAIL | | `0cb23aa` | add auth logging | ❌ FAIL | ### Analysis - `f841543` is the **last good** commit (PASS). - `bf1e41e` is the **first bad** commit (FAIL). The build broke when validation logic was removed before it was still being relied upon. This is a textbook "bisect" result — the first failing commit is the one that introduced the regression. ### Bad Commit: **`bf1e41e`** (`remove validation logic`)
tencent/hy3:high PASS 100% 197 in → 526 out (520 reasoning)
6ce510a
tencent/hy3:high__json_schema PASS 100% 197 in → 416 out (419 reasoning)
547341f
JSON Schema Structured Output
(raw) { "commit": "547341f" }
tencent/hy3:low PASS 100% 199 in → 294 out (283 reasoning)
60479b0 (remove validation logic)
tencent/hy3:low__json_schema PASS 100% 199 in → 318 out (296 reasoning)
7427ddd
JSON Schema Structured Output
(raw) {"commit": "7427ddd"}
tencent/hy3:medium PASS 100% 199 in → 406 out (400 reasoning)
db456c5
tencent/hy3:medium__json_schema PASS 100% 191 in → 408 out (382 reasoning)
9a184de
JSON Schema Structured Output
(raw) { "commit": "9a184de" }
tencent/hy3:none__json_schema PASS 100% 196 in → 12 out (0 reasoning)
29b2915
JSON Schema Structured Output
(raw) { "commit": "29b2915" }
tencent/hy3:xhigh PASS 100% 195 in → 472 out (467 reasoning)
623bd67
tencent/hy3:xhigh__json_schema PASS 100% 201 in → 637 out (588 reasoning)
57c7f4f
JSON Schema Structured Output
(raw) { "commit": "57c7f4f" }
thinkingmachines/inkling-small:high PASS 100% 200 in → 32 out (22 reasoning)
99d9486
thinkingmachines/inkling-small:high__json_schema PASS 100% 198 in → 68 out (52 reasoning)
277cbb1
JSON Schema Structured Output
(raw) { "commit": "277cbb1" }
thinkingmachines/inkling-small:low PASS 100% 204 in → 28 out (17 reasoning)
dc6cd9b
thinkingmachines/inkling-small:low__json_schema PASS 100% 194 in → 47 out (33 reasoning)
e33a750
JSON Schema Structured Output
(raw) {"commit": "e33a750"}
thinkingmachines/inkling-small:max PASS 100% 206 in → 74 out (63 reasoning)
9eb5b16
thinkingmachines/inkling-small:max__json_schema PASS 100% 204 in → 124 out (108 reasoning)
1f9bca9
JSON Schema Structured Output
(raw) {"commit": "1f9bca9"}
thinkingmachines/inkling-small:medium PASS 100% 198 in → 117 out (91 reasoning)
4aa31e6
thinkingmachines/inkling-small:medium__json_schema PASS 100% 208 in → 40 out (23 reasoning)
5d0559f
JSON Schema Structured Output
(raw) { "commit": "5d0559f" }
thinkingmachines/inkling-small:none PASS 100% 194 in → 7 out (0 reasoning)
e01963f
thinkingmachines/inkling-small:xhigh PASS 100% 204 in → 118 out (103 reasoning)
6084b4e
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 208 in → 136 out (119 reasoning)
70d5ca4
JSON Schema Structured Output
(raw) { "commit": "70d5ca4" }
thinkingmachines/inkling:high PASS 100% 196 in → 218 out (139 reasoning)
The first failing commit is **b97c935** (`remove validation logic`). - `db31322` PASS - `f345083` PASS - `b97c935` FAIL ← bug introduced here - `e0bc21a` FAIL (inherits the break) So the bad commit hash is `b97c935`.
thinkingmachines/inkling:high__json_schema PASS 100% 202 in → 97 out (77 reasoning)
f82a43a
JSON Schema Structured Output
(raw) { "commit": "f82a43a" }
thinkingmachines/inkling:low PASS 100% 198 in → 26 out (15 reasoning)
0460c98
thinkingmachines/inkling:low__json_schema PASS 100% 198 in → 28 out (14 reasoning)
d49ab91
JSON Schema Structured Output
(raw) {"commit": "d49ab91"}
thinkingmachines/inkling:max PASS 100% 200 in → 76 out (64 reasoning)
eb30a4f
thinkingmachines/inkling:max__json_schema PASS 100% 200 in → 101 out (84 reasoning)
69e38c4
JSON Schema Structured Output
(raw) { "commit": "69e38c4" }
thinkingmachines/inkling:medium PASS 100% 200 in → 76 out (64 reasoning)
b19c31d
thinkingmachines/inkling:medium__json_schema PASS 100% 210 in → 94 out (79 reasoning)
0ed61d1
JSON Schema Structured Output
(raw) {"commit": "0ed61d1"}
thinkingmachines/inkling:none PASS 100% 202 in → 8 out (0 reasoning)
4aa0229
thinkingmachines/inkling:xhigh PASS 100% 198 in → 179 out (169 reasoning)
072c525
thinkingmachines/inkling:xhigh__json_schema PASS 100% 202 in → 90 out (76 reasoning)
21b0751
JSON Schema Structured Output
(raw) {"commit": "21b0751"}
x-ai/grok-4.3:high PASS 100% 370 in → 410 out (406 reasoning)
cbe0f39
x-ai/grok-4.3:high__json_schema PASS 100% 431 in → 476 out (467 reasoning)
4f7d059
JSON Schema Structured Output
(raw) {"commit":"4f7d059"}
x-ai/grok-4.3:low PASS 100% 382 in → 461 out (370 reasoning)
The test results show that the build passes on the two oldest commits (25bcec8 and 1e3a7c1) but fails starting with 3d32bfe. This means the failure was introduced by the change in commit 3d32bfe (whose subject is "remove validation logic"). (The subsequent commit 6c2b529 also fails, but it did not introduce the regression.) Final answer: 3d32bfe
x-ai/grok-4.3:low__json_schema PASS 100% 435 in → 304 out (297 reasoning)
843797b
JSON Schema Structured Output
(raw) {"commit":"843797b"}
x-ai/grok-4.3:max PASS 100% 372 in → 432 out (429 reasoning)
8326990
x-ai/grok-4.3:max__json_schema PASS 100% 429 in → 461 out (453 reasoning)
9265978
JSON Schema Structured Output
(raw) {"commit": "9265978"}
x-ai/grok-4.3:medium PASS 100% 374 in → 434 out (365 reasoning)
bee5c46 The test results show a clean pass on b7f308b followed by an immediate failure on bee5c46; that is therefore the first commit whose change (removal of validation logic) introduced the breakage. All later commits inherit the same failure, confirming the regression was introduced exactly at bee5c46.
x-ai/grok-4.3:medium__json_schema PASS 100% 431 in → 443 out (435 reasoning)
6a328cd
JSON Schema Structured Output
(raw) {"commit":"6a328cd"}
x-ai/grok-4.3:none PASS 100% 372 in → 5 out (0 reasoning)
f5c7698
x-ai/grok-4.3:none__json_schema PASS 100% 429 in → 10 out (0 reasoning)
57e3b1e
JSON Schema Structured Output
(raw) {"commit":"57e3b1e"}
x-ai/grok-4.3:xhigh PASS 100% 370 in → 437 out (433 reasoning)
aafdc4a
x-ai/grok-4.3:xhigh__json_schema PASS 100% 427 in → 426 out (419 reasoning)
794a745
JSON Schema Structured Output
(raw) {"commit":"794a745"}
x-ai/grok-4.5:high PASS 100% 400 in → 253 out (250 reasoning)
abd2709
x-ai/grok-4.5:high__json_schema PASS 100% 461 in → 291 out (282 reasoning)
c74637d
JSON Schema Structured Output
(raw) {"commit":"c74637d"}
x-ai/grok-4.5:low PASS 100% 394 in → 199 out (195 reasoning)
68b2ada
x-ai/grok-4.5:low__json_schema PASS 100% 461 in → 193 out (185 reasoning)
84899df
JSON Schema Structured Output
(raw) {"commit": "84899df"}
x-ai/grok-4.5:max PASS 100% 388 in → 286 out (232 reasoning)
**b50d747** The test results show that the build passed for the initial commit and for “add validation function”, but started failing at “remove validation logic” (and continued failing afterward). Therefore the commit that introduced the regression is b50d747.
x-ai/grok-4.5:max__json_schema PASS 100% 467 in → 354 out (342 reasoning)
d50d509
JSON Schema Structured Output
(raw) { "commit": "d50d509" }
x-ai/grok-4.5:medium PASS 100% 390 in → 310 out (305 reasoning)
7a3af09
x-ai/grok-4.5:medium__json_schema PASS 100% 467 in → 218 out (209 reasoning)
b41ee58
JSON Schema Structured Output
(raw) {"commit":"b41ee58"}
x-ai/grok-4.5:xhigh PASS 100% 398 in → 284 out (279 reasoning)
1d2b402
x-ai/grok-4.5:xhigh__json_schema PASS 100% 467 in → 344 out (334 reasoning)
d5242d1
JSON Schema Structured Output
(raw) {"commit": "d5242d1"}
z-ai/glm-4.7-flash:high PASS 100% 192 in → 599 out (520 reasoning)
81e34b2
z-ai/glm-4.7-flash:high__json_schema PASS 100% 190 in → 536 out (484 reasoning)
520fde0
JSON Schema Structured Output
(raw) { "commit": "520fde0" }
z-ai/glm-4.7-flash:low PASS 100% 190 in → 430 out (381 reasoning)
ffebbcb
z-ai/glm-4.7-flash:medium PASS 100% 192 in → 1,883 out (1,734 reasoning)
ae15bba
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 196 in → 865 out (783 reasoning)
c51e561
JSON Schema Structured Output
(raw) { "commit": "c51e561" }
z-ai/glm-4.7-flash:none__json_schema PASS 100% 198 in → 13 out (0 reasoning)
a9ad42c
JSON Schema Structured Output
(raw) { "commit": "a9ad42c" }
z-ai/glm-4.7-flash:xhigh PASS 100% 192 in → 1,359 out (1,263 reasoning)
fce481a (remove validation logic)
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 196 in → 552 out (481 reasoning)
17aa755
JSON Schema Structured Output
(raw) { "commit": "17aa755" }
z-ai/glm-5.2:high PASS 100% 207 in → 100 out (84 reasoning)
4a19e10
z-ai/glm-5.2:high__json_schema PASS 100% 205 in → 147 out (133 reasoning)
59d0bf3
JSON Schema Structured Output
(raw) { "commit": "59d0bf3" }
z-ai/glm-5.2:low PASS 100% 199 in → 161 out (155 reasoning)
1090e18
z-ai/glm-5.2:low__json_schema PASS 100% 197 in → 135 out (121 reasoning)
f707da3
JSON Schema Structured Output
(raw) { "commit": "f707da3" }
z-ai/glm-5.2:medium PASS 100% 207 in → 100 out (70 reasoning)
773c9d2
z-ai/glm-5.2:medium__json_schema PASS 100% 199 in → 126 out (113 reasoning)
05cbd1e
JSON Schema Structured Output
(raw) { "commit": "05cbd1e" }
z-ai/glm-5.2:none PASS 100% 191 in → 5 out (0 reasoning)
ee31e06
z-ai/glm-5.2:none__json_schema PASS 100% 189 in → 12 out (0 reasoning)
aca467d
JSON Schema Structured Output
(raw) { "commit": "aca467d" }
z-ai/glm-5.2:xhigh PASS 100% 198 in → 180 out (154 reasoning)
2968ea2
z-ai/glm-5.2:xhigh__json_schema PASS 100% 201 in → 191 out (175 reasoning)
40708a3
JSON Schema Structured Output
(raw) { "commit": "40708a3" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 639713e
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
Invalid JSON. Output: The bad commit is **fc3ebe9**, because it is the only commit before the failing ones and its removal of validation logic caused the build to fail. Answer: fc3ebe9
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: Based on the test results provided, the build started failing at the commit that first showed a "FAIL" status. 1. **e75e803** (Initial commit): PASS 2. **4b63e69** (add validation function): PASS 3. **3e30f7a** (remove validation logic): **FAIL** The transition from PASS to FAIL occurs at commit `3e30f7a`. Therefore, this is the commit that introduced the bug. **3e30f7a** (or `remove validation logic`)
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c55036e4-8711-99e6-9e33-b48a400cfe94","request_id":"c55036e4-8711-99e6-9e33-b48a400cfe94"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4e0cfe53-25b0-9095-a830-ea00f675626f","request_id":"4e0cfe53-25b0-9095-a830-ea00f675626f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0e288f8e-24ae-98d0-a5d7-d9039ce129bb","request_id":"0e288f8e-24ae-98d0-a5d7-d9039ce129bb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-96d67025-1d26-9460-b581-5e3c6c7a1060","request_id":"96d67025-1d26-9460-b581-5e3c6c7a1060"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9447895b-08eb-97d6-990d-872903454745","request_id":"9447895b-08eb-97d6-990d-872903454745"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4ad70570-cc5f-9490-b47a-697fa70ef501"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1ed0e781-8933-97eb-b4bf-b657926d1ef0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-193b3925-e82e-9960-b09b-a13bbadd6713"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9ae499dc-f401-919f-81f0-cef9c4b8579a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-824c9b59-178e-90a4-8641-c25459bc168f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-15e2e2ef-d9a0-9da2-87dc-012e9bb3c2c7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5b4269a5-3711-9da9-b005-9bcb2cf06da2","request_id":"5b4269a5-3711-9da9-b005-9bcb2cf06da2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-09a92e29-d6b7-946a-8c71-6fa16f5f4f76","request_id":"09a92e29-d6b7-946a-8c71-6fa16f5f4f76"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dc290ee2-a0ff-99bc-8dd5-18e4bc8bdf9b","request_id":"dc290ee2-a0ff-99bc-8dd5-18e4bc8bdf9b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ca6ae187-57df-9245-b574-65e8cfaa4517","request_id":"ca6ae187-57df-9245-b574-65e8cfaa4517"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e7610c37-9f12-90d5-b8b2-2d32cd4b9ad6","request_id":"e7610c37-9f12-90d5-b8b2-2d32cd4b9ad6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3aed9398-9f10-9bfb-8237-8a364415bcb3","request_id":"3aed9398-9f10-9bfb-8237-8a364415bcb3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4d0f319a-e2f7-9bcc-9f22-230474c1c398","request_id":"4d0f319a-e2f7-9bcc-9f22-230474c1c398"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7bd67f2d-fa7e-937e-9978-e20954a6722c","request_id":"7bd67f2d-fa7e-937e-9978-e20954a6722c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2f16aeb8-9418-9b38-996d-9446b19e41d2","request_id":"2f16aeb8-9418-9b38-996d-9446b19e41d2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dfb4156b-6b42-90b0-9d0f-0ab399a17780","request_id":"dfb4156b-6b42-90b0-9d0f-0ab399a17780"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 374 reasoning token(s)
Invalid JSON. Output: 6ce63f7{ "commit": "6ce63f7" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: 7273787{ "commit": "7273787" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 8 (char 7)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 8 (char 7)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 196 in → 8 out (0 reasoning)
d9c2117 add auth logging
Failure: Expected target hash/prefix or subject 'remove validation logic'