Show merge commit and its parents
Tests ability to inspect a merge commit and identify its parents. Evaluates merge-commit structure comprehension.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'base' > shared.txt
  5. 05 git add shared.txt
  6. 06 git commit -m 'Base commit'
  7. 07 git checkout -b feature
  8. 08 echo 'feature' > feature.txt
  9. 09 git add feature.txt
  10. 10 git commit -m 'Feature work'
  11. 11 git checkout main
  12. 12 echo 'main change' > shared.txt
  13. 13 git add shared.txt
  14. 14 git commit -m 'Main line work'
  15. 15 git merge feature -m 'Merge feature branch'
Prompt
Using git show --merges -s --format=%P, how many parent commits does the merge commit have? Output ONLY the number, nothing else.
Expected
2

Scoped model quality, cost, API time, and token usage for git_show/f009.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 370 in → 29 out (17 reasoning)
2
anthropic/claude-fable-5:high__json_schema PASS 100% 567 in → 21 out (24 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-fable-5:low PASS 100% 368 in → 3 out (0 reasoning)
2
anthropic/claude-fable-5:low__json_schema PASS 100% 569 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-fable-5:max PASS 100% 360 in → 165 out (36 reasoning)
2
anthropic/claude-fable-5:max__json_schema PASS 100% 572 in → 72 out (32 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-fable-5:medium PASS 100% 362 in → 15 out (15 reasoning)
2
anthropic/claude-fable-5:medium__json_schema PASS 100% 581 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-fable-5:xhigh PASS 100% 360 in → 34 out (32 reasoning)
2
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 571 in → 40 out (33 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-haiku-4.5:high PASS 100% 325 in → 397 out (313 reasoning)
2
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 501 in → 316 out (252 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-haiku-4.5:low PASS 100% 333 in → 309 out (244 reasoning)
2
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 492 in → 472 out (373 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-haiku-4.5:medium PASS 100% 326 in → 270 out (224 reasoning)
2
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 495 in → 437 out (341 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-haiku-4.5:none PASS 100% 299 in → 5 out (0 reasoning)
2
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 459 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-haiku-4.5:xhigh PASS 100% 330 in → 216 out (162 reasoning)
2
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 490 in → 391 out (299 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.6:high PASS 100% 304 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:high__json_schema PASS 100% 469 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.6:low PASS 100% 302 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:low__json_schema PASS 100% 460 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.6:max PASS 100% 303 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:max__json_schema PASS 100% 469 in → 174 out (117 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.6:medium PASS 100% 301 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 468 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.6:none PASS 100% 303 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:none__json_schema PASS 100% 457 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.6:xhigh PASS 100% 301 in → 5 out (0 reasoning)
2
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 462 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.7:high PASS 100% 364 in → 6 out (0 reasoning)
2
anthropic/claude-opus-4.7:high__json_schema PASS 100% 583 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.7:low PASS 100% 366 in → 6 out (0 reasoning)
2
anthropic/claude-opus-4.7:low__json_schema PASS 100% 585 in → 11 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.7:max PASS 100% 362 in → 84 out (35 reasoning)
2
anthropic/claude-opus-4.7:max__json_schema PASS 100% 576 in → 66 out (36 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.7:medium PASS 100% 371 in → 6 out (0 reasoning)
2
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 579 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.7:none PASS 100% 368 in → 6 out (0 reasoning)
2
anthropic/claude-opus-4.7:none__json_schema PASS 100% 576 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.7:xhigh PASS 100% 373 in → 6 out (0 reasoning)
2
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 588 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.8:high PASS 100% 357 in → 36 out (29 reasoning)
2
anthropic/claude-opus-4.8:high__json_schema PASS 100% 560 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.8:low PASS 100% 374 in → 3 out (0 reasoning)
2
anthropic/claude-opus-4.8:low__json_schema PASS 100% 570 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.8:max PASS 100% 370 in → 202 out (52 reasoning)
2
anthropic/claude-opus-4.8:max__json_schema PASS 100% 574 in → 145 out (60 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-opus-4.8:medium PASS 100% 361 in → 3 out (0 reasoning)
2
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 574 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.8:none PASS 100% 364 in → 3 out (0 reasoning)
2
anthropic/claude-opus-4.8:none__json_schema PASS 100% 572 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-opus-4.8:xhigh PASS 100% 367 in → 35 out (21 reasoning)
2
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 580 in → 45 out (32 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-sonnet-4.6:high PASS 100% 308 in → 5 out (0 reasoning)
2
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 461 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-4.6:low PASS 100% 304 in → 5 out (0 reasoning)
2
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 465 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-4.6:max PASS 100% 306 in → 169 out (117 reasoning)
2
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 454 in → 163 out (117 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-4.6:medium PASS 100% 304 in → 5 out (0 reasoning)
2
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 464 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-4.6:none PASS 100% 300 in → 5 out (0 reasoning)
2
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 459 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 299 in → 77 out (34 reasoning)
2
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 463 in → 80 out (32 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-5:high PASS 100% 360 in → 3 out (0 reasoning)
2
anthropic/claude-sonnet-5:high__json_schema PASS 100% 575 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-sonnet-5:low PASS 100% 353 in → 3 out (0 reasoning)
2
anthropic/claude-sonnet-5:low__json_schema PASS 100% 571 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-5:medium PASS 100% 358 in → 3 out (0 reasoning)
2
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 569 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
anthropic/claude-sonnet-5:none PASS 100% 361 in → 3 out (0 reasoning)
2
anthropic/claude-sonnet-5:none__json_schema PASS 100% 572 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
anthropic/claude-sonnet-5:xhigh PASS 100% 360 in → 36 out (34 reasoning)
2
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 570 in → 39 out (29 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
arcee-ai/trinity-large-thinking:high PASS 100% 254 in → 562 out (559 reasoning)
2
arcee-ai/trinity-large-thinking:low PASS 100% 254 in → 504 out (505 reasoning)
2
arcee-ai/trinity-large-thinking:medium PASS 100% 256 in → 732 out (737 reasoning)
2
arcee-ai/trinity-large-thinking:xhigh PASS 100% 260 in → 704 out (708 reasoning)
2
arcee-ai/trinity-mini:high PASS 100% 256 in → 272 out (252 reasoning)
2
arcee-ai/trinity-mini:high__json_schema PASS 100% 258 in → 231 out (235 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
arcee-ai/trinity-mini:low PASS 100% 262 in → 255 out (244 reasoning)
2
arcee-ai/trinity-mini:low__json_schema PASS 100% 259 in → 258 out (246 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
arcee-ai/trinity-mini:medium PASS 100% 260 in → 209 out (198 reasoning)
2
arcee-ai/trinity-mini:medium__json_schema PASS 100% 257 in → 186 out (178 reasoning)
2
JSON Schema Structured Output
(raw) { "count":2 }
arcee-ai/trinity-mini:xhigh PASS 100% 256 in → 294 out (294 reasoning)
2
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 250 in → 221 out (224 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
deepseek/deepseek-v4-flash-0731:high PASS 100% 262 in → 87 out (65 reasoning)
2
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 332 in → 118 out (105 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
deepseek/deepseek-v4-flash-0731:low PASS 100% 265 in → 112 out (109 reasoning)
2
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 269 in → 117 out (87 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
deepseek/deepseek-v4-flash-0731:max PASS 100% 351 in → 181 out (177 reasoning)
2
deepseek/deepseek-v4-flash-0731:medium PASS 100% 263 in → 79 out (76 reasoning)
2
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 257 in → 100 out (75 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
deepseek/deepseek-v4-flash-0731:none PASS 100% 268 in → 2 out (0 reasoning)
2
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 265 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 260 in → 166 out (233 reasoning)
2
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 261 in → 87 out (56 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
deepseek/deepseek-v4-flash:high PASS 100% 261 in → 100 out (97 reasoning)
2
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 349 in → 6 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
deepseek/deepseek-v4-flash:low PASS 100% 263 in → 88 out (81 reasoning)
2
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 269 in → 221 out (210 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
deepseek/deepseek-v4-flash:medium PASS 100% 260 in → 128 out (100 reasoning)
2
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 262 in → 154 out (145 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
deepseek/deepseek-v4-flash:none PASS 100% 265 in → 1 out (0 reasoning)
2
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 255 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
deepseek/deepseek-v4-flash:xhigh PASS 100% 336 in → 247 out (217 reasoning)
2
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 342 in → 143 out (135 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
deepseek/deepseek-v4-pro:high PASS 100% 257 in → 141 out (138 reasoning)
2
deepseek/deepseek-v4-pro:low PASS 100% 257 in → 144 out (142 reasoning)
2
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 450 in → 89 out (81 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
deepseek/deepseek-v4-pro:medium PASS 100% 258 in → 260 out (254 reasoning)
2
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 259 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
deepseek/deepseek-v4-pro:none PASS 100% 259 in → 1 out (0 reasoning)
2
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 442 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
deepseek/deepseek-v4-pro:xhigh PASS 100% 339 in → 223 out (220 reasoning)
2
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 516 in → 477 out (469 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3-flash-preview:high PASS 100% 323 in → 349 out (348 reasoning)
2
google/gemini-3-flash-preview:high__json_schema PASS 100% 318 in → 446 out (441 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3-flash-preview:low PASS 100% 320 in → 309 out (308 reasoning)
2
google/gemini-3-flash-preview:low__json_schema PASS 100% 382 in → 435 out (430 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3-flash-preview:medium PASS 100% 320 in → 402 out (401 reasoning)
2
google/gemini-3-flash-preview:medium__json_schema PASS 100% 374 in → 427 out (422 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3-flash-preview:xhigh PASS 100% 323 in → 361 out (360 reasoning)
2
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 380 in → 729 out (724 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.1-flash-lite-preview:high PASS 100% 317 in → 376 out (375 reasoning)
2
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 379 in → 220 out (210 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemini-3.1-flash-lite-preview:low PASS 100% 322 in → 137 out (136 reasoning)
2
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 379 in → 166 out (156 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 326 in → 208 out (207 reasoning)
2
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 319 in → 179 out (169 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 326 in → 630 out (629 reasoning)
2
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 372 in → 360 out (350 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemini-3.1-pro-preview:high PASS 100% 319 in → 348 out (347 reasoning)
2
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 376 in → 449 out (444 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.1-pro-preview:low PASS 100% 320 in → 162 out (161 reasoning)
2
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 370 in → 227 out (221 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.1-pro-preview:medium PASS 100% 321 in → 125 out (124 reasoning)
2
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 380 in → 126 out (120 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.1-pro-preview:xhigh PASS 100% 328 in → 142 out (141 reasoning)
2
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 385 in → 248 out (242 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.5-flash-lite:high PASS 100% 317 in → 401 out (400 reasoning)
2
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 321 in → 407 out (401 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.5-flash-lite:low PASS 100% 324 in → 1 out (0 reasoning)
2
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 325 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemini-3.5-flash-lite:max PASS 100% 323 in → 409 out (408 reasoning)
2
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 376 in → 493 out (487 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.5-flash-lite:medium PASS 100% 322 in → 298 out (297 reasoning)
2
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 375 in → 337 out (332 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.5-flash-lite:xhigh PASS 100% 326 in → 322 out (321 reasoning)
2
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 380 in → 334 out (328 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.5-flash:high PASS 100% 324 in → 591 out (590 reasoning)
2
google/gemini-3.5-flash:high__json_schema PASS 100% 372 in → 417 out (412 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.5-flash:low PASS 100% 325 in → 145 out (144 reasoning)
2
google/gemini-3.5-flash:low__json_schema PASS 100% 375 in → 123 out (117 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.5-flash:medium PASS 100% 314 in → 439 out (438 reasoning)
2
google/gemini-3.5-flash:medium__json_schema PASS 100% 365 in → 275 out (270 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.5-flash:xhigh PASS 100% 324 in → 398 out (397 reasoning)
2
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 375 in → 398 out (393 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.6-flash:high PASS 100% 328 in → 345 out (344 reasoning)
2
google/gemini-3.6-flash:high__json_schema PASS 100% 326 in → 334 out (329 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.6-flash:low PASS 100% 325 in → 138 out (137 reasoning)
2
google/gemini-3.6-flash:low__json_schema PASS 100% 322 in → 134 out (129 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.6-flash:max PASS 100% 321 in → 419 out (418 reasoning)
2
google/gemini-3.6-flash:max__json_schema PASS 100% 378 in → 270 out (264 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
google/gemini-3.6-flash:medium PASS 100% 323 in → 359 out (358 reasoning)
2
google/gemini-3.6-flash:medium__json_schema PASS 100% 323 in → 235 out (230 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemini-3.6-flash:xhigh PASS 100% 323 in → 379 out (378 reasoning)
2
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 377 in → 480 out (475 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemma-4-26b-a4b-it:high PASS 100% 331 in → 706 out (525 reasoning)
2
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 330 in → 565 out (345 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-26b-a4b-it:low PASS 100% 331 in → 416 out (341 reasoning)
2
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 343 in → 479 out (304 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-26b-a4b-it:medium PASS 100% 334 in → 582 out (402 reasoning)
2
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 335 in → 471 out (316 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-26b-a4b-it:none PASS 100% 338 in → 2 out (0 reasoning)
2
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 331 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 338 in → 535 out (350 reasoning)
2
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 330 in → 631 out (458 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-31b-it:high PASS 100% 341 in → 365 out (363 reasoning)
2
google/gemma-4-31b-it:high__json_schema PASS 100% 339 in → 408 out (232 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-31b-it:low PASS 100% 336 in → 410 out (257 reasoning)
2
google/gemma-4-31b-it:low__json_schema PASS 100% 342 in → 321 out (315 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
google/gemma-4-31b-it:medium PASS 100% 340 in → 440 out (268 reasoning)
2
google/gemma-4-31b-it:medium__json_schema PASS 100% 340 in → 386 out (231 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-31b-it:none PASS 100% 335 in → 2 out (0 reasoning)
2
google/gemma-4-31b-it:none__json_schema PASS 100% 337 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
google/gemma-4-31b-it:xhigh PASS 100% 333 in → 415 out (227 reasoning)
2
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 346 in → 398 out (240 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
ibm-granite/granite-4.1-8b PASS 100% 279 in → 2 out
2
ibm-granite/granite-4.1-8b__json_schema PASS 100% 290 in → 10 out
2
JSON Schema Structured Output
(raw) { "count": 2 }
inclusionai/ling-2.6-flash PASS 100% 339 in → 2 out
2
2
JSON Schema Structured Output
(raw) {"count": 2}
minimax/minimax-m2.5:high PASS 100% 298 in → 233 out (201 reasoning)
2
minimax/minimax-m2.5:high__json_schema PASS 100% 284 in → 291 out (275 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m2.5:low__json_schema PASS 100% 316 in → 144 out (133 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m2.5:medium PASS 100% 286 in → 131 out (128 reasoning)
2
minimax/minimax-m2.5:medium__json_schema PASS 100% 290 in → 212 out (205 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m2.5:xhigh PASS 100% 300 in → 327 out (324 reasoning)
2
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 286 in → 245 out (234 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m2.7:high PASS 100% 292 in → 128 out (126 reasoning)
2
minimax/minimax-m2.7:high__json_schema PASS 100% 419 in → 324 out (317 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
minimax/minimax-m2.7:low PASS 100% 287 in → 191 out (189 reasoning)
2
minimax/minimax-m2.7:low__json_schema PASS 100% 417 in → 243 out (235 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
minimax/minimax-m2.7:medium PASS 100% 293 in → 229 out (226 reasoning)
2
minimax/minimax-m2.7:xhigh PASS 100% 293 in → 345 out (341 reasoning)
2
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 290 in → 11 out (10 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m3:high PASS 100% 424 in → 157 out (149 reasoning)
2
minimax/minimax-m3:high__json_schema PASS 100% 417 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m3:low PASS 100% 421 in → 112 out (113 reasoning)
2
minimax/minimax-m3:low__json_schema PASS 100% 376 in → 12 out (9 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
minimax/minimax-m3:medium PASS 100% 423 in → 47 out (30 reasoning)
2
minimax/minimax-m3:xhigh PASS 100% 426 in → 110 out (102 reasoning)
2
minimax/minimax-m3:xhigh__json_schema PASS 100% 424 in → 9 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
mistralai/devstral-2512 PASS 100% 314 in → 2 out
2
mistralai/devstral-2512__json_schema PASS 100% 307 in → 7 out
2
JSON Schema Structured Output
(raw) {"count": 2}
mistralai/mistral-medium-3-5:high PASS 100% 329 in → 482 out (464 reasoning)
2
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 311 in → 357 out (223 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
mistralai/mistral-medium-3-5:low PASS 100% 322 in → 271 out (243 reasoning)
2
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 311 in → 362 out (252 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
mistralai/mistral-medium-3-5:medium PASS 100% 322 in → 393 out (349 reasoning)
2
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 313 in → 455 out (408 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
mistralai/mistral-medium-3-5:none PASS 100% 322 in → 2 out (0 reasoning)
2
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 307 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
mistralai/mistral-medium-3-5:xhigh PASS 100% 322 in → 343 out (317 reasoning)
2
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 310 in → 336 out (261 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
moonshotai/kimi-k2.6:high PASS 100% 263 in → 333 out (330 reasoning)
2
moonshotai/kimi-k2.6:high__json_schema PASS 100% 255 in → 759 out (232 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2}
moonshotai/kimi-k2.6:low PASS 100% 261 in → 275 out (272 reasoning)
2
moonshotai/kimi-k2.6:low__json_schema PASS 100% 251 in → 1,675 out (167 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
moonshotai/kimi-k2.6:medium PASS 100% 253 in → 312 out (302 reasoning)
2
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 254 in → 240 out (230 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
moonshotai/kimi-k2.6:xhigh PASS 100% 259 in → 288 out (286 reasoning)
2
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 252 in → 184 out (179 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
moonshotai/kimi-k2.7-code:high PASS 100% 257 in → 129 out (126 reasoning)
2
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 262 in → 146 out (122 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
moonshotai/kimi-k2.7-code:low PASS 100% 266 in → 82 out (75 reasoning)
2
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 267 in → 125 out (114 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
moonshotai/kimi-k2.7-code:medium PASS 100% 258 in → 149 out (146 reasoning)
2
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 259 in → 106 out (102 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 253 in → 154 out (151 reasoning)
2
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 251 in → 103 out (86 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
moonshotai/kimi-k3:high PASS 100% 328 in → 176 out (160 reasoning)
2
moonshotai/kimi-k3:high__json_schema PASS 100% 339 in → 145 out (129 reasoning)
2
moonshotai/kimi-k3:low PASS 100% 332 in → 75 out (59 reasoning)
2
moonshotai/kimi-k3:low__json_schema PASS 100% 330 in → 148 out (132 reasoning)
2
moonshotai/kimi-k3:max PASS 100% 335 in → 153 out (137 reasoning)
2
moonshotai/kimi-k3:max__json_schema PASS 100% 327 in → 110 out (94 reasoning)
2
moonshotai/kimi-k3:medium PASS 100% 335 in → 140 out (124 reasoning)
2
moonshotai/kimi-k3:medium__json_schema PASS 100% 336 in → 157 out (141 reasoning)
2
moonshotai/kimi-k3:xhigh PASS 100% 330 in → 177 out (161 reasoning)
2
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 328 in → 248 out (232 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 316 in → 233 out (222 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 327 in → 209 out (142 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2 }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 332 in → 555 out (492 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 331 in → 174 out (149 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 326 in → 197 out (171 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 328 in → 417 out (407 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 330 in → 2 out (0 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 331 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 326 in → 248 out (256 reasoning)
2
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 323 in → 268 out (192 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 332 in → 139 out (112 reasoning)
2
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 324 in → 187 out (167 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 333 in → 110 out (69 reasoning)
2
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 326 in → 135 out (105 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 324 in → 148 out (123 reasoning)
2
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 330 in → 323 out (261 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 324 in → 2 out (0 reasoning)
2
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 324 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 326 in → 441 out (366 reasoning)
2
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 326 in → 113 out (96 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
openai/gpt-5.4-mini:high PASS 100% 257 in → 76 out (69 reasoning)
2
openai/gpt-5.4-mini:high__json_schema PASS 100% 289 in → 65 out (50 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-mini:low PASS 100% 262 in → 30 out (23 reasoning)
2
openai/gpt-5.4-mini:low__json_schema PASS 100% 289 in → 41 out (26 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-mini:medium PASS 100% 260 in → 52 out (45 reasoning)
2
openai/gpt-5.4-mini:medium__json_schema PASS 100% 294 in → 88 out (73 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-mini:none PASS 100% 260 in → 5 out (0 reasoning)
2
openai/gpt-5.4-mini:none__json_schema PASS 100% 290 in → 19 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-mini:xhigh PASS 100% 257 in → 126 out (119 reasoning)
2
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 292 in → 228 out (213 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-nano:high PASS 100% 263 in → 71 out (64 reasoning)
2
openai/gpt-5.4-nano:high__json_schema PASS 100% 300 in → 89 out (74 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-nano:low PASS 100% 260 in → 62 out (55 reasoning)
2
openai/gpt-5.4-nano:low__json_schema PASS 100% 286 in → 54 out (39 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-nano:medium PASS 100% 255 in → 103 out (96 reasoning)
2
openai/gpt-5.4-nano:medium__json_schema PASS 100% 294 in → 78 out (63 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-nano:none PASS 100% 258 in → 5 out (0 reasoning)
2
openai/gpt-5.4-nano:none__json_schema PASS 100% 292 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4-nano:xhigh PASS 100% 259 in → 191 out (184 reasoning)
2
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 285 in → 658 out (643 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4:high PASS 100% 264 in → 74 out (67 reasoning)
2
openai/gpt-5.4:high__json_schema PASS 100% 303 in → 429 out (414 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4:low PASS 100% 262 in → 45 out (38 reasoning)
2
openai/gpt-5.4:low__json_schema PASS 100% 289 in → 48 out (33 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4:medium PASS 100% 258 in → 68 out (61 reasoning)
2
openai/gpt-5.4:medium__json_schema PASS 100% 293 in → 68 out (53 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4:none PASS 100% 259 in → 5 out (0 reasoning)
2
openai/gpt-5.4:none__json_schema PASS 100% 289 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.4:xhigh PASS 100% 256 in → 135 out (128 reasoning)
2
openai/gpt-5.4:xhigh__json_schema PASS 100% 289 in → 90 out (75 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.5:high PASS 100% 262 in → 86 out (79 reasoning)
2
openai/gpt-5.5:high__json_schema PASS 100% 288 in → 208 out (193 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.5:low PASS 100% 258 in → 45 out (38 reasoning)
2
openai/gpt-5.5:low__json_schema PASS 100% 289 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.5:medium PASS 100% 254 in → 65 out (58 reasoning)
2
openai/gpt-5.5:medium__json_schema PASS 100% 288 in → 140 out (125 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.5:none PASS 100% 259 in → 5 out (0 reasoning)
2
openai/gpt-5.5:none__json_schema PASS 100% 292 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.5:xhigh PASS 100% 265 in → 173 out (166 reasoning)
2
openai/gpt-5.5:xhigh__json_schema PASS 100% 288 in → 197 out (182 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:high PASS 100% 260 in → 5 out (0 reasoning)
2
openai/gpt-5.6-luna:high__json_schema PASS 100% 291 in → 103 out (88 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:low PASS 100% 256 in → 5 out (0 reasoning)
2
openai/gpt-5.6-luna:low__json_schema PASS 100% 286 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:max PASS 100% 254 in → 73 out (66 reasoning)
2
openai/gpt-5.6-luna:max__json_schema PASS 100% 290 in → 247 out (232 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:medium PASS 100% 263 in → 5 out (0 reasoning)
2
openai/gpt-5.6-luna:medium__json_schema PASS 100% 295 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:none PASS 100% 262 in → 5 out (0 reasoning)
2
openai/gpt-5.6-luna:none__json_schema PASS 100% 300 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-luna:xhigh PASS 100% 263 in → 62 out (55 reasoning)
2
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 290 in → 80 out (65 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:high PASS 100% 259 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:high__json_schema PASS 100% 295 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:low PASS 100% 269 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:low__json_schema PASS 100% 292 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:max PASS 100% 262 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:max__json_schema PASS 100% 292 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:medium PASS 100% 255 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:medium__json_schema PASS 100% 293 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:none PASS 100% 263 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:none__json_schema PASS 100% 291 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-sol:xhigh PASS 100% 261 in → 5 out (0 reasoning)
2
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 291 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:high PASS 100% 255 in → 5 out (0 reasoning)
2
openai/gpt-5.6-terra:high__json_schema PASS 100% 291 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:low PASS 100% 262 in → 5 out (0 reasoning)
2
openai/gpt-5.6-terra:low__json_schema PASS 100% 299 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:max PASS 100% 252 in → 55 out (48 reasoning)
2
openai/gpt-5.6-terra:max__json_schema PASS 100% 288 in → 70 out (55 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:medium PASS 100% 258 in → 5 out (0 reasoning)
2
openai/gpt-5.6-terra:medium__json_schema PASS 100% 292 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:none PASS 100% 262 in → 5 out (0 reasoning)
2
openai/gpt-5.6-terra:none__json_schema PASS 100% 287 in → 13 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-5.6-terra:xhigh PASS 100% 262 in → 5 out (0 reasoning)
2
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 292 in → 52 out (37 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-oss-120b:high PASS 100% 322 in → 247 out (248 reasoning)
2
openai/gpt-oss-120b:high__json_schema PASS 100% 321 in → 450 out (414 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-oss-120b:low PASS 100% 310 in → 28 out (19 reasoning)
2
openai/gpt-oss-120b:low__json_schema PASS 100% 327 in → 103 out (76 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
openai/gpt-oss-120b:medium PASS 100% 315 in → 145 out (113 reasoning)
2
openai/gpt-oss-120b:medium__json_schema PASS 100% 314 in → 111 out (96 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-oss-120b:xhigh PASS 100% 307 in → 226 out (214 reasoning)
2
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 314 in → 295 out (285 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
openai/gpt-oss-20b:high PASS 100% 318 in → 262 out (218 reasoning)
2
openai/gpt-oss-20b:high__json_schema PASS 100% 307 in → 375 out (313 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
openai/gpt-oss-20b:low PASS 100% 320 in → 37 out (18 reasoning)
2
openai/gpt-oss-20b:low__json_schema PASS 100% 368 in → 82 out (64 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
openai/gpt-oss-20b:medium PASS 100% 319 in → 220 out (184 reasoning)
2
openai/gpt-oss-20b:xhigh PASS 100% 311 in → 293 out (260 reasoning)
2
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 304 in → 778 out (679 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-m.1:high PASS 100% 324 in → 352 out (348 reasoning)
2
poolside/laguna-m.1:high__json_schema PASS 100% 333 in → 344 out (333 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-m.1:low PASS 100% 320 in → 253 out (249 reasoning)
2
poolside/laguna-m.1:low__json_schema PASS 100% 329 in → 451 out (440 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-m.1:medium PASS 100% 328 in → 374 out (370 reasoning)
2
poolside/laguna-m.1:medium__json_schema PASS 100% 321 in → 356 out (345 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-m.1:none PASS 100% 325 in → 3 out (0 reasoning)
2
poolside/laguna-m.1:none__json_schema PASS 100% 317 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-m.1:xhigh PASS 100% 328 in → 385 out (381 reasoning)
2
poolside/laguna-m.1:xhigh__json_schema PASS 100% 328 in → 274 out (263 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs-2.1:high PASS 100% 331 in → 402 out (400 reasoning)
2
poolside/laguna-xs-2.1:high__json_schema PASS 100% 326 in → 397 out (385 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs-2.1:low PASS 100% 324 in → 339 out (337 reasoning)
2
poolside/laguna-xs-2.1:low__json_schema PASS 100% 328 in → 353 out (341 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs-2.1:medium PASS 100% 321 in → 461 out (459 reasoning)
2
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 331 in → 437 out (425 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 327 in → 11 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs-2.1:xhigh PASS 100% 317 in → 382 out (380 reasoning)
2
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 327 in → 322 out (310 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
poolside/laguna-xs.2:high PASS 100% 357 in → 232 out (228 reasoning)
2
poolside/laguna-xs.2:high__json_schema PASS 100% 364 in → 248 out (238 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
poolside/laguna-xs.2:low PASS 100% 364 in → 257 out (254 reasoning)
2
poolside/laguna-xs.2:low__json_schema PASS 100% 359 in → 213 out (205 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
poolside/laguna-xs.2:medium PASS 100% 358 in → 247 out (243 reasoning)
2
poolside/laguna-xs.2:medium__json_schema PASS 100% 367 in → 293 out (284 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
poolside/laguna-xs.2:none PASS 100% 363 in → 3 out (0 reasoning)
2
poolside/laguna-xs.2:none__json_schema PASS 100% 364 in → 7 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
poolside/laguna-xs.2:xhigh PASS 100% 364 in → 375 out (372 reasoning)
2
qwen/qwen3.6-27b:high PASS 100% 321 in → 650 out (525 reasoning)
2
qwen/qwen3.6-27b:low PASS 100% 318 in → 383 out (318 reasoning)
2
qwen/qwen3.6-27b:low__json_schema PASS 100% 325 in → 264 out (211 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-27b:medium PASS 100% 324 in → 378 out (327 reasoning)
2
qwen/qwen3.6-27b:medium__json_schema PASS 100% 326 in → 442 out (355 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-27b:none PASS 100% 324 in → 2 out (0 reasoning)
2
qwen/qwen3.6-27b:xhigh PASS 100% 327 in → 568 out (433 reasoning)
2
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 319 in → 889 out (1 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-35b-a3b:high PASS 100% 320 in → 565 out (479 reasoning)
2
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 323 in → 355 out (280 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
qwen/qwen3.6-35b-a3b:low PASS 100% 325 in → 446 out (440 reasoning)
2
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 327 in → 589 out (454 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-35b-a3b:medium PASS 100% 330 in → 470 out (367 reasoning)
2
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 316 in → 441 out (334 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-35b-a3b:none PASS 100% 322 in → 2 out (0 reasoning)
2
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 322 in → 11 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 319 in → 791 out (785 reasoning)
2
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 326 in → 439 out (355 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
qwen/qwen3.6-flash:high PASS 100% 328 in → 563 out (557 reasoning)
2
qwen/qwen3.6-flash:low PASS 100% 322 in → 384 out (378 reasoning)
2
qwen/qwen3.6-flash:medium PASS 100% 330 in → 375 out (369 reasoning)
2
qwen/qwen3.6-flash:none PASS 100% 332 in → 1 out (0 reasoning)
2
qwen/qwen3.6-flash:xhigh PASS 100% 326 in → 359 out (354 reasoning)
2
qwen/qwen3.7-flash:high PASS 100% 315 in → 365 out (359 reasoning)
2
qwen/qwen3.7-flash:low PASS 100% 318 in → 368 out (363 reasoning)
2
qwen/qwen3.7-flash:max PASS 100% 325 in → 308 out (302 reasoning)
2
qwen/qwen3.7-flash:medium PASS 100% 312 in → 438 out (432 reasoning)
2
qwen/qwen3.7-flash:none PASS 100% 330 in → 1 out (0 reasoning)
2
qwen/qwen3.7-flash:none__json_schema PASS 100% 326 in → 1 out (0 reasoning)
2
qwen/qwen3.7-flash:xhigh PASS 100% 318 in → 300 out (294 reasoning)
2
qwen/qwen3.7-max:high PASS 100% 319 in → 224 out (218 reasoning)
2
qwen/qwen3.7-max:low PASS 100% 328 in → 303 out (297 reasoning)
2
qwen/qwen3.7-max:medium PASS 100% 322 in → 257 out (252 reasoning)
2
qwen/qwen3.7-max:none PASS 100% 325 in → 1 out (0 reasoning)
2
qwen/qwen3.7-max:xhigh PASS 100% 322 in → 297 out (291 reasoning)
2
qwen/qwen3.7-plus:high PASS 100% 319 in → 388 out (382 reasoning)
2
qwen/qwen3.7-plus:low PASS 100% 322 in → 408 out (402 reasoning)
2
qwen/qwen3.7-plus:medium PASS 100% 325 in → 211 out (205 reasoning)
2
qwen/qwen3.7-plus:none PASS 100% 323 in → 1 out (0 reasoning)
2
qwen/qwen3.7-plus:xhigh PASS 100% 328 in → 207 out (201 reasoning)
2
tencent/hy3:high PASS 100% 272 in → 369 out (366 reasoning)
2
tencent/hy3:high__json_schema PASS 100% 266 in → 315 out (241 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
tencent/hy3:low PASS 100% 267 in → 230 out (227 reasoning)
2
tencent/hy3:low__json_schema PASS 100% 260 in → 213 out (190 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
tencent/hy3:medium PASS 100% 265 in → 252 out (249 reasoning)
2
tencent/hy3:medium__json_schema PASS 100% 264 in → 284 out (243 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
tencent/hy3:none PASS 100% 271 in → 2 out (0 reasoning)
2
tencent/hy3:none__json_schema PASS 100% 264 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
tencent/hy3:xhigh PASS 100% 267 in → 206 out (203 reasoning)
2
tencent/hy3:xhigh__json_schema PASS 100% 271 in → 356 out (298 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
thinkingmachines/inkling-small:high PASS 100% 266 in → 102 out (95 reasoning)
2
thinkingmachines/inkling-small:high__json_schema PASS 100% 270 in → 102 out (88 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
thinkingmachines/inkling-small:low PASS 100% 264 in → 40 out (30 reasoning)
2
thinkingmachines/inkling-small:low__json_schema PASS 100% 261 in → 19 out (7 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
thinkingmachines/inkling-small:max PASS 100% 268 in → 78 out (63 reasoning)
2
thinkingmachines/inkling-small:max__json_schema PASS 100% 271 in → 179 out (166 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
thinkingmachines/inkling-small:medium PASS 100% 267 in → 86 out (58 reasoning)
2
thinkingmachines/inkling-small:medium__json_schema PASS 100% 272 in → 108 out (97 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
thinkingmachines/inkling-small:xhigh PASS 100% 263 in → 150 out (122 reasoning)
2
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 267 in → 125 out (113 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
thinkingmachines/inkling:high PASS 100% 272 in → 119 out (111 reasoning)
2
thinkingmachines/inkling:low PASS 100% 262 in → 53 out (45 reasoning)
2
thinkingmachines/inkling:max PASS 100% 270 in → 107 out (100 reasoning)
2
thinkingmachines/inkling:max__json_schema PASS 100% 265 in → 115 out (101 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
thinkingmachines/inkling:medium PASS 100% 272 in → 84 out (77 reasoning)
2
thinkingmachines/inkling:xhigh PASS 100% 272 in → 90 out (83 reasoning)
2
thinkingmachines/inkling:xhigh__json_schema PASS 100% 263 in → 90 out (76 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
x-ai/grok-4.3:high PASS 100% 437 in → 345 out (344 reasoning)
2
x-ai/grok-4.3:high__json_schema PASS 100% 488 in → 512 out (507 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.3:low__json_schema PASS 100% 495 in → 406 out (401 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.3:max PASS 100% 433 in → 409 out (408 reasoning)
2
x-ai/grok-4.3:max__json_schema PASS 100% 488 in → 484 out (479 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.3:medium__json_schema PASS 100% 493 in → 544 out (539 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.3:none PASS 100% 438 in → 1 out (0 reasoning)
2
x-ai/grok-4.3:none__json_schema PASS 100% 482 in → 5 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.3:xhigh PASS 100% 437 in → 312 out (311 reasoning)
2
x-ai/grok-4.3:xhigh__json_schema PASS 100% 489 in → 465 out (460 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.5:high PASS 100% 464 in → 308 out (307 reasoning)
2
x-ai/grok-4.5:high__json_schema PASS 100% 524 in → 503 out (498 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.5:low PASS 100% 459 in → 143 out (142 reasoning)
2
x-ai/grok-4.5:low__json_schema PASS 100% 527 in → 286 out (281 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.5:max PASS 100% 459 in → 332 out (331 reasoning)
2
x-ai/grok-4.5:max__json_schema PASS 100% 530 in → 525 out (520 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
x-ai/grok-4.5:medium PASS 100% 467 in → 297 out (296 reasoning)
2
x-ai/grok-4.5:medium__json_schema PASS 100% 525 in → 424 out (415 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
x-ai/grok-4.5:xhigh PASS 100% 461 in → 430 out (429 reasoning)
2
x-ai/grok-4.5:xhigh__json_schema PASS 100% 536 in → 287 out (282 reasoning)
2
JSON Schema Structured Output
(raw) {"count":2}
z-ai/glm-4.7-flash:high PASS 100% 271 in → 767 out (639 reasoning)
2
z-ai/glm-4.7-flash:low PASS 100% 273 in → 633 out (540 reasoning)
2
z-ai/glm-4.7-flash:low__json_schema PASS 100% 275 in → 563 out (455 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
z-ai/glm-4.7-flash:medium PASS 100% 267 in → 602 out (481 reasoning)
2
z-ai/glm-4.7-flash:none PASS 100% 274 in → 2 out (0 reasoning)
2
z-ai/glm-4.7-flash:none__json_schema PASS 100% 266 in → 10 out (0 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
z-ai/glm-4.7-flash:xhigh PASS 100% 264 in → 644 out (622 reasoning)
2
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 265 in → 483 out (399 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
z-ai/glm-5.2:high PASS 100% 273 in → 99 out (96 reasoning)
2
z-ai/glm-5.2:low PASS 100% 279 in → 126 out (111 reasoning)
2
z-ai/glm-5.2:low__json_schema PASS 100% 272 in → 163 out (154 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
z-ai/glm-5.2:medium PASS 100% 273 in → 159 out (156 reasoning)
2
z-ai/glm-5.2:medium__json_schema PASS 100% 270 in → 197 out (188 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
z-ai/glm-5.2:none PASS 100% 267 in → 2 out (0 reasoning)
2
z-ai/glm-5.2:none__json_schema PASS 100% 275 in → 8 out (0 reasoning)
2
JSON Schema Structured Output
(raw) {"count": 2}
z-ai/glm-5.2:xhigh PASS 100% 265 in → 179 out (145 reasoning)
2
z-ai/glm-5.2:xhigh__json_schema PASS 100% 276 in → 370 out (285 reasoning)
2
JSON Schema Structured Output
(raw) { "count": 2 }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
liquid/lfm-2-24b-a2b FAIL 0% 275 in → 2 out
1
Failure: Expected '2', got '1'
Invalid structured output. Output: 1
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
minimax/minimax-m2.5:low FAIL 0% 289 in → 232 out (213 reasoning)
(empty output)
Failure: Expected '2', got ''
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:medium__json_schema FAIL 0% 372 in → 13 out (10 reasoning)
1
JSON Schema Structured Output
(raw) { "count": 1 }
Failure: Expected '2', got '1'
openai/gpt-oss-20b:medium__json_schema FAIL 0% 314 in → 3,231 out (100 reasoning)
1
JSON Schema Structured Output
(raw) { "count" : 1 }
Failure: Expected '2', got '1'
poolside/laguna-xs-2.1:none FAIL 0% 329 in → 4 out (0 reasoning)
2 </think>2
Failure: Expected '2', got '2 </think>2'
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 370 in → 254 out (246 reasoning)
0
JSON Schema Structured Output
(raw) {"count": 0}
Failure: Expected '2', got '0'
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
qwen/qwen3.6-27b:none__json_schema FAIL 0% 324 in → 7 out (0 reasoning)
1
JSON Schema Structured Output
(raw) {"count": 1}
Failure: Expected '2', got '1'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-76124290-0954-9047-92d1-fd6f36a37313","request_id":"76124290-0954-9047-92d1-fd6f36a37313"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-63d13b21-3f6f-909d-8bb3-2efd8c3228f3","request_id":"63d13b21-3f6f-909d-8bb3-2efd8c3228f3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-be7baa46-ae56-9477-b227-f5febc533d0c","request_id":"be7baa46-ae56-9477-b227-f5febc533d0c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-daf8a151-7a16-9c18-a4bf-5cbe7471db62","request_id":"daf8a151-7a16-9c18-a4bf-5cbe7471db62"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1ec321d1-fbac-9e76-9f16-1680a9031ebd","request_id":"1ec321d1-fbac-9e76-9f16-1680a9031ebd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b51d7b11-02e3-939e-8263-8887500c28c6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4639579a-719d-911a-a95f-545ce99a071b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c1759cc4-c499-905c-9786-4d41b03d995c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0c110ee0-ee52-9383-bd0c-a48025dddc2b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5e719447-3243-97dd-99f3-9a66c9ef69a8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-32916a14-4a0e-9c31-b38c-7b41a79a04aa","request_id":"32916a14-4a0e-9c31-b38c-7b41a79a04aa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a5cd4b54-d397-9c9f-8e16-1b34e6ccc25f","request_id":"a5cd4b54-d397-9c9f-8e16-1b34e6ccc25f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f5b8f0c6-eb59-9c8a-aa28-85e4948d3b69","request_id":"f5b8f0c6-eb59-9c8a-aa28-85e4948d3b69"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e5faf708-fd15-90eb-aa93-2db6b82bdf49","request_id":"e5faf708-fd15-90eb-aa93-2db6b82bdf49"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3161e0c0-5246-97d1-b7a1-2e3a842f6df6","request_id":"3161e0c0-5246-97d1-b7a1-2e3a842f6df6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7b18c3d6-24af-95cd-bc85-8f02a70abf7f","request_id":"7b18c3d6-24af-95cd-bc85-8f02a70abf7f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3a2a4924-a9dd-9a1b-bf4b-8859a93a79c7","request_id":"3a2a4924-a9dd-9a1b-bf4b-8859a93a79c7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c1f53d9e-70c9-950c-b241-8642827bb367","request_id":"c1f53d9e-70c9-950c-b241-8642827bb367"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e541902d-cba0-910c-948f-fb1aff23f6a0","request_id":"e541902d-cba0-910c-948f-fb1aff23f6a0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-58e6445e-35f1-9462-b181-96f10147c557","request_id":"58e6445e-35f1-9462-b181-96f10147c557"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 269 in → 4 out (0 reasoning)
2<|end_message|>
Failure: Expected '2', got '2<|end_message|>'
Invalid JSON. Output: 2{ "count": 2 }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
thinkingmachines/inkling:none FAIL 0% 270 in → 4 out (0 reasoning)
2<|end_message|>
Failure: Expected '2', got '2<|end_message|>'
Invalid JSON. Output: 2{"count": 2}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
x-ai/grok-4.3:low FAIL 0% 434 in → 441 out (346 reasoning)
2 The provided log output shows one merge commit (8fcd61ad3f870854726c56ace4217e0334ab3fef). Its Merge: header explicitly lists two parent SHAs (0f007b5 and 093bc93). Git records each parent SHA on that line, so the merge commit has exactly two parents. The format specifier %P simply emits those parent SHAs; counting the resulting tokens therefore yields the integer 2.
Failure: Expected '2', got '2 The provided log output shows one merge commit (8fcd61ad3f870854726c56ace4217e0334ab3fef). Its Merge: header explicitly lists two parent SHAs (0f007b5 and 093bc93). Git records each parent SHA on that line, so the merge commit has exactly two parents. The format specifier %P simply emits those parent SHAs; counting the resulting tokens therefore yields the integer 2.'
x-ai/grok-4.3:medium FAIL 0% 433 in → 833 out (700 reasoning)
2 The provided repository state contains exactly one merge commit (`ed5d04225f729ac8998b7487b01f7f24ff463160`). Its header line explicitly lists two parent SHA-1 values (`Merge: f32a994 f200625`), therefore the commit has two parents. Running the cited command filters to merge commits only (`--merges`), suppresses the patch (`-s`), and emits the raw parent list (`--format=%P`). The resulting output therefore contains exactly two hashes separated by a space, confirming the parent count. No other merge commit exists in the supplied history, so the numeric answer is 2.
Failure: Expected '2', got '2 The provided repository state contains exactly one merge commit (`ed5d04225f729ac8998b7487b01f7f24ff463160`). Its header line explicitly lists two parent SHA-1 values (`Merge: f32a994 f200625`), therefore the commit has two parents. Running the cited command filters to merge commits only (`--merges`), suppresses the patch (`-s`), and emits the raw parent list (`--format=%P`). The resulting output therefore contains exactly two hashes separated by a space, confirming the parent count. No other merge commit exists in the supplied history, so the numeric answer is 2.'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid structured output. Output: 2
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)