f004 — git_show

← git_show
Show annotated tag with tagger info and message
Tests ability to inspect an annotated tag with tagger info and message. Evaluates annotated-tag comprehension.
medium git-show annotated-tag tagger message
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'releaser@company.com'
  3. 03 git config user.name 'Release Bot'
  4. 04 echo 'stable code' > main.py
  5. 05 git add main.py
  6. 06 git commit -m 'Stable release'
  7. 07 git tag -a v2.0.0 -m 'Major release: stable API'
Prompt
Using git show, what is the tagger email for the annotated tag 'v2.0.0'? Output ONLY the email, nothing else.
Expected
releaser@company.com
Fixture Metrics

Scoped model quality, cost, API time, and token usage for git_show/f004.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high__json_schema PASS 100% 732 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-fable-5:low PASS 100% 528 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-fable-5:low__json_schema PASS 100% 728 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-fable-5:max PASS 100% 514 in → 56 out (27 reasoning)
releaser@company.com
anthropic/claude-fable-5:max__json_schema PASS 100% 718 in → 61 out (28 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-fable-5:medium PASS 100% 532 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-fable-5:medium__json_schema PASS 100% 724 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-fable-5:xhigh PASS 100% 528 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 722 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-haiku-4.5:high PASS 100% 471 in → 168 out (131 reasoning)
releaser@company.com
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 629 in → 294 out (273 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-haiku-4.5:low PASS 100% 469 in → 133 out (99 reasoning)
releaser@company.com
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 633 in → 273 out (221 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-haiku-4.5:medium PASS 100% 477 in → 149 out (110 reasoning)
releaser@company.com
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 637 in → 458 out (430 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-haiku-4.5:none PASS 100% 450 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 599 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 477 in → 159 out (119 reasoning)
releaser@company.com
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 633 in → 257 out (226 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:high PASS 100% 444 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:high__json_schema PASS 100% 604 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:low PASS 100% 444 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:low__json_schema PASS 100% 608 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:max PASS 100% 436 in → 83 out (50 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:max__json_schema PASS 100% 606 in → 87 out (50 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:medium PASS 100% 440 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 602 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:none PASS 100% 442 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:none__json_schema PASS 100% 610 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.6:xhigh PASS 100% 446 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 608 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:high PASS 100% 533 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:high__json_schema PASS 100% 733 in → 18 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:low PASS 100% 531 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:low__json_schema PASS 100% 731 in → 18 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:max PASS 100% 527 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:max__json_schema PASS 100% 735 in → 18 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:medium PASS 100% 535 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 731 in → 19 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:none PASS 100% 539 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:none__json_schema PASS 100% 733 in → 19 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.7:xhigh PASS 100% 525 in → 11 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 741 in → 18 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:high PASS 100% 522 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:high__json_schema PASS 100% 726 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:low PASS 100% 526 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:low__json_schema PASS 100% 730 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:max PASS 100% 524 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:max__json_schema PASS 100% 736 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:medium PASS 100% 534 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 730 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:none PASS 100% 526 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:none__json_schema PASS 100% 724 in → 15 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-opus-4.8:xhigh PASS 100% 516 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 722 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:high PASS 100% 436 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 598 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:low PASS 100% 442 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 610 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:max PASS 100% 448 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 606 in → 85 out (48 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:medium PASS 100% 450 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 604 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:none PASS 100% 440 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 608 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 444 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 608 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-5:high PASS 100% 530 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-5:high__json_schema PASS 100% 722 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-sonnet-5:low PASS 100% 528 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-5:low__json_schema PASS 100% 722 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
anthropic/claude-sonnet-5:medium PASS 100% 520 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 736 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-sonnet-5:none PASS 100% 520 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-5:none__json_schema PASS 100% 722 in → 15 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
anthropic/claude-sonnet-5:xhigh PASS 100% 532 in → 9 out (0 reasoning)
releaser@company.com
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 736 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
arcee-ai/trinity-large-thinking:high PASS 100% 381 in → 336 out (327 reasoning)
releaser@company.com
arcee-ai/trinity-large-thinking:low PASS 100% 383 in → 442 out (436 reasoning)
releaser@company.com
arcee-ai/trinity-large-thinking:medium PASS 100% 383 in → 342 out (334 reasoning)
releaser@company.com
arcee-ai/trinity-large-thinking:xhigh PASS 100% 389 in → 371 out (362 reasoning)
releaser@company.com
arcee-ai/trinity-mini:high PASS 100% 389 in → 315 out (326 reasoning)
releaser@company.com
arcee-ai/trinity-mini:high__json_schema PASS 100% 389 in → 271 out (275 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email":"releaser@company.com" }
arcee-ai/trinity-mini:low PASS 100% 383 in → 331 out (337 reasoning)
releaser@company.com
arcee-ai/trinity-mini:low__json_schema PASS 100% 389 in → 275 out (283 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
arcee-ai/trinity-mini:medium PASS 100% 391 in → 265 out (265 reasoning)
releaser@company.com
arcee-ai/trinity-mini:medium__json_schema PASS 100% 383 in → 278 out (275 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
arcee-ai/trinity-mini:xhigh PASS 100% 391 in → 279 out (277 reasoning)
releaser@company.com
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 387 in → 248 out (250 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 388 in → 59 out (51 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 392 in → 67 out (48 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 392 in → 61 out (46 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 394 in → 106 out (90 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 471 in → 75 out (63 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 467 in → 35 out (22 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 384 in → 93 out (80 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 388 in → 66 out (50 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 384 in → 7 out (0 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 394 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 465 in → 81 out (74 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 386 in → 77 out (58 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash:high PASS 100% 390 in → 68 out (60 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 388 in → 70 out (52 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash:low PASS 100% 382 in → 58 out (50 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 388 in → 79 out (62 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash:medium PASS 100% 388 in → 71 out (63 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 382 in → 121 out (97 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash:none PASS 100% 380 in → 6 out (0 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 390 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 467 in → 105 out (97 reasoning)
releaser@company.com
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 471 in → 194 out (166 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-pro:high PASS 100% 386 in → 72 out (64 reasoning)
releaser@company.com
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 380 in → 16 out (14 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-pro:low PASS 100% 390 in → 176 out (148 reasoning)
releaser@company.com
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 392 in → 80 out (65 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-pro:medium PASS 100% 394 in → 146 out (138 reasoning)
releaser@company.com
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 384 in → 15 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
deepseek/deepseek-v4-pro:none PASS 100% 392 in → 7 out (0 reasoning)
releaser@company.com
deepseek/deepseek-v4-pro:xhigh PASS 100% 388 in → 83 out (68 reasoning)
releaser@company.com
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 649 in → 300 out (287 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3-flash-preview:high PASS 100% 480 in → 232 out (225 reasoning)
releaser@company.com
google/gemini-3-flash-preview:high__json_schema PASS 100% 533 in → 409 out (398 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3-flash-preview:low PASS 100% 474 in → 197 out (190 reasoning)
releaser@company.com
google/gemini-3-flash-preview:low__json_schema PASS 100% 533 in → 237 out (226 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3-flash-preview:medium PASS 100% 482 in → 184 out (177 reasoning)
releaser@company.com
google/gemini-3-flash-preview:medium__json_schema PASS 100% 533 in → 181 out (165 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemini-3-flash-preview:xhigh PASS 100% 476 in → 206 out (199 reasoning)
releaser@company.com
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 487 in → 238 out (227 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 480 in → 374 out (367 reasoning)
releaser@company.com
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 481 in → 425 out (414 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 491 in → 324 out (317 reasoning)
releaser@company.com
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 483 in → 128 out (111 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 482 in → 212 out (205 reasoning)
releaser@company.com
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 483 in → 207 out (195 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 478 in → 383 out (376 reasoning)
releaser@company.com
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 479 in → 238 out (227 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.1-pro-preview:high PASS 100% 480 in → 283 out (276 reasoning)
releaser@company.com
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 533 in → 384 out (372 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.1-pro-preview:low PASS 100% 478 in → 217 out (210 reasoning)
releaser@company.com
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 535 in → 85 out (73 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.1-pro-preview:medium PASS 100% 480 in → 304 out (297 reasoning)
releaser@company.com
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 523 in → 92 out (81 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 476 in → 235 out (228 reasoning)
releaser@company.com
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 535 in → 376 out (365 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.5-flash-lite:high PASS 100% 488 in → 256 out (249 reasoning)
releaser@company.com
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 535 in → 366 out (354 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.5-flash-lite:low PASS 100% 484 in → 7 out (0 reasoning)
releaser@company.com
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 543 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemini-3.5-flash-lite:max PASS 100% 472 in → 290 out (283 reasoning)
releaser@company.com
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 481 in → 488 out (477 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.5-flash-lite:medium PASS 100% 485 in → 176 out (169 reasoning)
releaser@company.com
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 481 in → 279 out (267 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 484 in → 329 out (322 reasoning)
releaser@company.com
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 541 in → 389 out (378 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.5-flash:high PASS 100% 480 in → 252 out (245 reasoning)
releaser@company.com
google/gemini-3.5-flash:high__json_schema PASS 100% 537 in → 218 out (207 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.5-flash:low PASS 100% 480 in → 83 out (76 reasoning)
releaser@company.com
google/gemini-3.5-flash:low__json_schema PASS 100% 541 in → 88 out (76 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemini-3.5-flash:medium PASS 100% 474 in → 264 out (257 reasoning)
releaser@company.com
google/gemini-3.5-flash:medium__json_schema PASS 100% 535 in → 175 out (158 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemini-3.5-flash:xhigh PASS 100% 482 in → 326 out (319 reasoning)
releaser@company.com
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 539 in → 375 out (364 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.6-flash:high PASS 100% 486 in → 154 out (147 reasoning)
releaser@company.com
google/gemini-3.6-flash:high__json_schema PASS 100% 541 in → 243 out (232 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.6-flash:low PASS 100% 481 in → 50 out (43 reasoning)
releaser@company.com
google/gemini-3.6-flash:low__json_schema PASS 100% 483 in → 11 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.6-flash:max PASS 100% 479 in → 302 out (295 reasoning)
releaser@company.com
google/gemini-3.6-flash:max__json_schema PASS 100% 487 in → 213 out (202 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.6-flash:medium PASS 100% 483 in → 135 out (128 reasoning)
releaser@company.com
google/gemini-3.6-flash:medium__json_schema PASS 100% 539 in → 207 out (196 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemini-3.6-flash:xhigh PASS 100% 484 in → 264 out (257 reasoning)
releaser@company.com
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 487 in → 254 out (243 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
google/gemma-4-26b-a4b-it:high PASS 100% 499 in → 306 out (235 reasoning)
releaser@company.com
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 491 in → 257 out (200 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemma-4-26b-a4b-it:low PASS 100% 492 in → 260 out (252 reasoning)
releaser@company.com
google/gemma-4-26b-a4b-it:medium PASS 100% 502 in → 318 out (220 reasoning)
releaser@company.com
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 494 in → 356 out (247 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemma-4-26b-a4b-it:none PASS 100% 489 in → 8 out (0 reasoning)
releaser@company.com
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 489 in → 18 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 494 in → 317 out (235 reasoning)
releaser@company.com
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 491 in → 421 out (289 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
google/gemma-4-31b-it:high PASS 100% 497 in → 251 out (243 reasoning)
releaser@company.com
google/gemma-4-31b-it:high__json_schema PASS 100% 495 in → 134 out (99 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemma-4-31b-it:low PASS 100% 490 in → 162 out (107 reasoning)
releaser@company.com
google/gemma-4-31b-it:low__json_schema PASS 100% 490 in → 124 out (74 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemma-4-31b-it:medium PASS 100% 494 in → 235 out (179 reasoning)
releaser@company.com
google/gemma-4-31b-it:medium__json_schema PASS 100% 500 in → 133 out (86 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemma-4-31b-it:none PASS 100% 491 in → 8 out (0 reasoning)
releaser@company.com
google/gemma-4-31b-it:none__json_schema PASS 100% 493 in → 13 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
google/gemma-4-31b-it:xhigh PASS 100% 498 in → 146 out (117 reasoning)
releaser@company.com
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 498 in → 219 out (167 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
ibm-granite/granite-4.1-8b PASS 100% 420 in → 7 out
releaser@company.com
ibm-granite/granite-4.1-8b__json_schema PASS 100% 420 in → 14 out
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
inclusionai/ling-2.6-flash PASS 100% 495 in → 7 out
releaser@company.com
inclusionai/ling-2.6-flash__json_schema PASS 100% 497 in → 14 out
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com"}
liquid/lfm-2-24b-a2b PASS 100% 403 in → 7 out
releaser@company.com
minimax/minimax-m2.5:high PASS 100% 413 in → 246 out (238 reasoning)
releaser@company.com
minimax/minimax-m2.5:high__json_schema PASS 100% 421 in → 133 out (108 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m2.5:low PASS 100% 423 in → 118 out (108 reasoning)
releaser@company.com
minimax/minimax-m2.5:low__json_schema PASS 100% 450 in → 128 out (96 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m2.5:medium PASS 100% 421 in → 191 out (183 reasoning)
releaser@company.com
minimax/minimax-m2.5:medium__json_schema PASS 100% 421 in → 193 out (178 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m2.5:xhigh PASS 100% 462 in → 111 out (103 reasoning)
releaser@company.com
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 421 in → 138 out (111 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m2.7:high PASS 100% 423 in → 121 out (114 reasoning)
releaser@company.com
minimax/minimax-m2.7:high__json_schema PASS 100% 552 in → 340 out (328 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
minimax/minimax-m2.7:low PASS 100% 417 in → 199 out (190 reasoning)
releaser@company.com
minimax/minimax-m2.7:low__json_schema PASS 100% 552 in → 242 out (230 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
minimax/minimax-m2.7:medium PASS 100% 417 in → 203 out (190 reasoning)
releaser@company.com
minimax/minimax-m2.7:medium__json_schema PASS 100% 546 in → 290 out (278 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
minimax/minimax-m2.7:xhigh PASS 100% 413 in → 250 out (285 reasoning)
releaser@company.com
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 556 in → 401 out (389 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
minimax/minimax-m3:high PASS 100% 559 in → 61 out (52 reasoning)
releaser@company.com
minimax/minimax-m3:high__json_schema PASS 100% 542 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m3:low PASS 100% 557 in → 102 out (85 reasoning)
releaser@company.com
minimax/minimax-m3:low__json_schema PASS 100% 501 in → 14 out (13 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m3:medium PASS 100% 553 in → 62 out (53 reasoning)
releaser@company.com
minimax/minimax-m3:medium__json_schema PASS 100% 548 in → 15 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
minimax/minimax-m3:xhigh PASS 100% 555 in → 93 out (69 reasoning)
releaser@company.com
minimax/minimax-m3:xhigh__json_schema PASS 100% 542 in → 15 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
mistralai/devstral-2512 PASS 100% 465 in → 7 out
releaser@company.com
mistralai/devstral-2512__json_schema PASS 100% 465 in → 12 out
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
mistralai/mistral-medium-3-5:high PASS 100% 481 in → 193 out (178 reasoning)
releaser@company.com
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 463 in → 235 out (220 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
mistralai/mistral-medium-3-5:low PASS 100% 479 in → 254 out (234 reasoning)
releaser@company.com
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 455 in → 163 out (150 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
mistralai/mistral-medium-3-5:medium PASS 100% 471 in → 187 out (177 reasoning)
releaser@company.com
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 459 in → 232 out (204 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
mistralai/mistral-medium-3-5:none PASS 100% 469 in → 7 out (0 reasoning)
releaser@company.com
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 455 in → 12 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 469 in → 276 out (249 reasoning)
releaser@company.com
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 453 in → 216 out (189 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
moonshotai/kimi-k2.6:high PASS 100% 379 in → 147 out (139 reasoning)
releaser@company.com
moonshotai/kimi-k2.6:high__json_schema PASS 100% 387 in → 122 out (107 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.6:low PASS 100% 397 in → 167 out (159 reasoning)
releaser@company.com
moonshotai/kimi-k2.6:low__json_schema PASS 100% 386 in → 291 out (139 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.6:medium PASS 100% 387 in → 275 out (295 reasoning)
releaser@company.com
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 391 in → 151 out (136 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.6:xhigh PASS 100% 391 in → 157 out (149 reasoning)
releaser@company.com
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 392 in → 698 out (171 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.7-code:high PASS 100% 387 in → 96 out (88 reasoning)
releaser@company.com
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 389 in → 143 out (128 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.7-code:low PASS 100% 383 in → 146 out (138 reasoning)
releaser@company.com
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 397 in → 56 out (39 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
moonshotai/kimi-k2.7-code:medium PASS 100% 391 in → 93 out (87 reasoning)
releaser@company.com
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 475 in → 121 out (109 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 385 in → 14 out (5 reasoning)
releaser@company.com
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 467 in → 88 out (76 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
moonshotai/kimi-k3:high PASS 100% 468 in → 117 out (96 reasoning)
releaser@company.com
moonshotai/kimi-k3:high__json_schema PASS 100% 458 in → 83 out (62 reasoning)
releaser@company.com
moonshotai/kimi-k3:low PASS 100% 466 in → 102 out (81 reasoning)
releaser@company.com
moonshotai/kimi-k3:low__json_schema PASS 100% 466 in → 102 out (81 reasoning)
releaser@company.com
moonshotai/kimi-k3:max PASS 100% 464 in → 73 out (52 reasoning)
releaser@company.com
moonshotai/kimi-k3:max__json_schema PASS 100% 464 in → 86 out (65 reasoning)
releaser@company.com
moonshotai/kimi-k3:medium PASS 100% 474 in → 118 out (97 reasoning)
releaser@company.com
moonshotai/kimi-k3:medium__json_schema PASS 100% 458 in → 84 out (63 reasoning)
releaser@company.com
moonshotai/kimi-k3:xhigh PASS 100% 464 in → 77 out (56 reasoning)
releaser@company.com
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 454 in → 77 out (56 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 480 in → 234 out (214 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 482 in → 58 out (39 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 476 in → 108 out (107 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 482 in → 98 out (81 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 474 in → 163 out (166 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 482 in → 97 out (74 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 476 in → 7 out (0 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 484 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 482 in → 117 out (108 reasoning)
releaser@company.com
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 486 in → 105 out (89 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 480 in → 156 out (137 reasoning)
releaser@company.com
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 476 in → 78 out (62 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 484 in → 70 out (60 reasoning)
releaser@company.com
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 480 in → 69 out (57 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 480 in → 96 out (89 reasoning)
releaser@company.com
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 472 in → 60 out (45 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 484 in → 7 out (0 reasoning)
releaser@company.com
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 476 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 470 in → 70 out (61 reasoning)
releaser@company.com
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 472 in → 123 out (106 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-5.4-mini:high PASS 100% 389 in → 59 out (47 reasoning)
releaser@company.com
openai/gpt-5.4-mini:high__json_schema PASS 100% 420 in → 206 out (187 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-mini:low PASS 100% 387 in → 21 out (9 reasoning)
releaser@company.com
openai/gpt-5.4-mini:low__json_schema PASS 100% 416 in → 38 out (19 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-mini:medium PASS 100% 389 in → 48 out (36 reasoning)
releaser@company.com
openai/gpt-5.4-mini:medium__json_schema PASS 100% 418 in → 203 out (184 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-mini:none PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4-mini:none__json_schema PASS 100% 408 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-mini:xhigh PASS 100% 381 in → 104 out (92 reasoning)
releaser@company.com
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 412 in → 2,089 out (2,070 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-nano:high PASS 100% 377 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4-nano:high__json_schema PASS 100% 416 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-nano:low PASS 100% 389 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4-nano:low__json_schema PASS 100% 414 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-nano:medium PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4-nano:medium__json_schema PASS 100% 422 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-nano:none PASS 100% 389 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4-nano:none__json_schema PASS 100% 414 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4-nano:xhigh PASS 100% 385 in → 68 out (56 reasoning)
releaser@company.com
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 426 in → 86 out (67 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4:high PASS 100% 385 in → 56 out (44 reasoning)
releaser@company.com
openai/gpt-5.4:high__json_schema PASS 100% 420 in → 63 out (44 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4:low PASS 100% 389 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4:low__json_schema PASS 100% 424 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4:medium PASS 100% 385 in → 22 out (10 reasoning)
releaser@company.com
openai/gpt-5.4:medium__json_schema PASS 100% 414 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4:none PASS 100% 395 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.4:none__json_schema PASS 100% 424 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.4:xhigh PASS 100% 387 in → 190 out (178 reasoning)
releaser@company.com
openai/gpt-5.4:xhigh__json_schema PASS 100% 422 in → 424 out (405 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.5:high PASS 100% 385 in → 52 out (40 reasoning)
releaser@company.com
openai/gpt-5.5:high__json_schema PASS 100% 420 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.5:low PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.5:low__json_schema PASS 100% 414 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.5:medium PASS 100% 393 in → 22 out (10 reasoning)
releaser@company.com
openai/gpt-5.5:medium__json_schema PASS 100% 422 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.5:none PASS 100% 379 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.5:none__json_schema PASS 100% 432 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.5:xhigh PASS 100% 387 in → 77 out (65 reasoning)
releaser@company.com
openai/gpt-5.5:xhigh__json_schema PASS 100% 428 in → 195 out (176 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:high PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-luna:high__json_schema PASS 100% 428 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:low PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-luna:low__json_schema PASS 100% 416 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:max PASS 100% 387 in → 38 out (26 reasoning)
releaser@company.com
openai/gpt-5.6-luna:max__json_schema PASS 100% 422 in → 501 out (482 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:medium PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-luna:medium__json_schema PASS 100% 420 in → 29 out (10 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:none PASS 100% 381 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-luna:none__json_schema PASS 100% 422 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-luna:xhigh PASS 100% 395 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 426 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:high PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:high__json_schema PASS 100% 414 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:low PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:low__json_schema PASS 100% 418 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:max PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:max__json_schema PASS 100% 412 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:medium PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:medium__json_schema PASS 100% 422 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:none PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:none__json_schema PASS 100% 416 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-sol:xhigh PASS 100% 391 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 428 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:high PASS 100% 393 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:high__json_schema PASS 100% 416 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:low PASS 100% 385 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:low__json_schema PASS 100% 412 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:max PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:max__json_schema PASS 100% 418 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:medium PASS 100% 391 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:medium__json_schema PASS 100% 420 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:none PASS 100% 383 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:none__json_schema PASS 100% 416 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-5.6-terra:xhigh PASS 100% 387 in → 10 out (0 reasoning)
releaser@company.com
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 426 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-oss-120b:high PASS 100% 448 in → 123 out (106 reasoning)
releaser@company.com
openai/gpt-oss-120b:high__json_schema PASS 100% 442 in → 119 out (96 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-120b:low PASS 100% 448 in → 26 out (11 reasoning)
releaser@company.com
openai/gpt-oss-120b:low__json_schema PASS 100% 441 in → 33 out (9 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-120b:medium PASS 100% 450 in → 96 out (81 reasoning)
releaser@company.com
openai/gpt-oss-120b:medium__json_schema PASS 100% 450 in → 85 out (55 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-120b:xhigh PASS 100% 450 in → 361 out (291 reasoning)
releaser@company.com
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 444 in → 210 out (184 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
openai/gpt-oss-20b:high PASS 100% 446 in → 267 out (238 reasoning)
releaser@company.com
openai/gpt-oss-20b:high__json_schema PASS 100% 450 in → 204 out (183 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-20b:low PASS 100% 448 in → 25 out (9 reasoning)
releaser@company.com
openai/gpt-oss-20b:low__json_schema PASS 100% 433 in → 60 out (32 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-20b:medium PASS 100% 444 in → 89 out (65 reasoning)
releaser@company.com
openai/gpt-oss-20b:medium__json_schema PASS 100% 446 in → 85 out (55 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
openai/gpt-oss-20b:xhigh PASS 100% 451 in → 220 out (184 reasoning)
releaser@company.com
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 443 in → 357 out (332 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email":"releaser@company.com" }
poolside/laguna-m.1:high PASS 100% 466 in → 319 out (312 reasoning)
releaser@company.com
poolside/laguna-m.1:high__json_schema PASS 100% 470 in → 322 out (307 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email" :"releaser@company.com"}
poolside/laguna-m.1:low PASS 100% 478 in → 349 out (343 reasoning)
releaser@company.com
poolside/laguna-m.1:low__json_schema PASS 100% 478 in → 283 out (267 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-m.1:medium PASS 100% 480 in → 238 out (230 reasoning)
releaser@company.com
poolside/laguna-m.1:medium__json_schema PASS 100% 472 in → 478 out (462 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-m.1:none PASS 100% 480 in → 8 out (0 reasoning)
releaser@company.com
poolside/laguna-m.1:none__json_schema PASS 100% 480 in → 12 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-m.1:xhigh PASS 100% 480 in → 277 out (269 reasoning)
releaser@company.com
poolside/laguna-m.1:xhigh__json_schema PASS 100% 474 in → 327 out (311 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-xs-2.1:high PASS 100% 480 in → 389 out (385 reasoning)
releaser@company.com
poolside/laguna-xs-2.1:high__json_schema PASS 100% 484 in → 242 out (225 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-xs-2.1:low PASS 100% 476 in → 234 out (228 reasoning)
releaser@company.com
poolside/laguna-xs-2.1:low__json_schema PASS 100% 484 in → 361 out (349 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs-2.1:medium PASS 100% 486 in → 288 out (285 reasoning)
releaser@company.com
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 476 in → 256 out (239 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-xs-2.1:none PASS 100% 482 in → 6 out (0 reasoning)
releaser@company.com
poolside/laguna-xs-2.1:none__json_schema PASS 100% 480 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
poolside/laguna-xs-2.1:xhigh PASS 100% 480 in → 256 out (250 reasoning)
releaser@company.com
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 484 in → 300 out (288 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs.2:high PASS 100% 511 in → 199 out (191 reasoning)
releaser@company.com
poolside/laguna-xs.2:high__json_schema PASS 100% 513 in → 149 out (137 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs.2:low PASS 100% 523 in → 147 out (139 reasoning)
releaser@company.com
poolside/laguna-xs.2:low__json_schema PASS 100% 513 in → 215 out (203 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs.2:medium PASS 100% 515 in → 136 out (128 reasoning)
releaser@company.com
poolside/laguna-xs.2:medium__json_schema PASS 100% 507 in → 159 out (147 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs.2:none PASS 100% 519 in → 8 out (0 reasoning)
releaser@company.com
poolside/laguna-xs.2:none__json_schema PASS 100% 513 in → 12 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
poolside/laguna-xs.2:xhigh PASS 100% 521 in → 135 out (127 reasoning)
releaser@company.com
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 519 in → 339 out (328 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
qwen/qwen3.6-27b:high PASS 100% 466 in → 255 out (220 reasoning)
releaser@company.com
qwen/qwen3.6-27b:high__json_schema PASS 100% 474 in → 284 out (246 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-27b:low PASS 100% 464 in → 177 out (145 reasoning)
releaser@company.com
qwen/qwen3.6-27b:medium PASS 100% 472 in → 173 out (1 reasoning)
releaser@company.com
qwen/qwen3.6-27b:medium__json_schema PASS 100% 474 in → 191 out (148 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-27b:none PASS 100% 478 in → 7 out (0 reasoning)
releaser@company.com
qwen/qwen3.6-27b:none__json_schema PASS 100% 476 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-27b:xhigh PASS 100% 468 in → 241 out (230 reasoning)
releaser@company.com
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 474 in → 640 out (543 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
qwen/qwen3.6-35b-a3b:high PASS 100% 474 in → 219 out (208 reasoning)
releaser@company.com
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 468 in → 316 out (276 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-35b-a3b:low PASS 100% 470 in → 284 out (250 reasoning)
releaser@company.com
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 476 in → 464 out (412 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 474 in → 532 out (483 reasoning)
releaser@company.com
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 472 in → 204 out (156 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-35b-a3b:none PASS 100% 476 in → 7 out (0 reasoning)
releaser@company.com
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 478 in → 17 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 474 in → 212 out (169 reasoning)
releaser@company.com
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 460 in → 244 out (206 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
qwen/qwen3.6-flash:high PASS 100% 468 in → 336 out (325 reasoning)
releaser@company.com
qwen/qwen3.6-flash:low PASS 100% 478 in → 700 out (689 reasoning)
releaser@company.com
qwen/qwen3.6-flash:medium PASS 100% 474 in → 232 out (221 reasoning)
releaser@company.com
qwen/qwen3.6-flash:none PASS 100% 476 in → 6 out (0 reasoning)
releaser@company.com
qwen/qwen3.6-flash:xhigh PASS 100% 482 in → 214 out (203 reasoning)
releaser@company.com
qwen/qwen3.7-flash:high PASS 100% 470 in → 335 out (324 reasoning)
releaser@company.com
qwen/qwen3.7-flash:low PASS 100% 470 in → 213 out (202 reasoning)
releaser@company.com
qwen/qwen3.7-flash:max PASS 100% 474 in → 197 out (186 reasoning)
releaser@company.com
qwen/qwen3.7-flash:medium PASS 100% 472 in → 255 out (244 reasoning)
releaser@company.com
qwen/qwen3.7-flash:none PASS 100% 478 in → 6 out (0 reasoning)
releaser@company.com
qwen/qwen3.7-flash:xhigh PASS 100% 470 in → 188 out (177 reasoning)
releaser@company.com
qwen/qwen3.7-max:high PASS 100% 470 in → 140 out (129 reasoning)
releaser@company.com
qwen/qwen3.7-max:low PASS 100% 476 in → 109 out (98 reasoning)
releaser@company.com
qwen/qwen3.7-max:medium PASS 100% 478 in → 168 out (157 reasoning)
releaser@company.com
qwen/qwen3.7-max:none PASS 100% 480 in → 6 out (0 reasoning)
releaser@company.com
qwen/qwen3.7-max:xhigh PASS 100% 474 in → 111 out (100 reasoning)
releaser@company.com
qwen/qwen3.7-plus:high PASS 100% 474 in → 161 out (150 reasoning)
releaser@company.com
qwen/qwen3.7-plus:low PASS 100% 466 in → 140 out (129 reasoning)
releaser@company.com
qwen/qwen3.7-plus:medium PASS 100% 470 in → 456 out (445 reasoning)
releaser@company.com
qwen/qwen3.7-plus:none PASS 100% 478 in → 6 out (0 reasoning)
releaser@company.com
qwen/qwen3.7-plus:xhigh PASS 100% 480 in → 350 out (339 reasoning)
releaser@company.com
tencent/hy3:high PASS 100% 391 in → 150 out (142 reasoning)
releaser@company.com
tencent/hy3:high__json_schema PASS 100% 395 in → 177 out (142 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
tencent/hy3:low PASS 100% 393 in → 157 out (149 reasoning)
releaser@company.com
tencent/hy3:low__json_schema PASS 100% 387 in → 128 out (96 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
tencent/hy3:medium PASS 100% 387 in → 226 out (218 reasoning)
releaser@company.com
tencent/hy3:medium__json_schema PASS 100% 395 in → 107 out (84 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
tencent/hy3:none PASS 100% 392 in → 7 out (0 reasoning)
releaser@company.com
tencent/hy3:none__json_schema PASS 100% 394 in → 16 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
tencent/hy3:xhigh PASS 100% 393 in → 217 out (209 reasoning)
releaser@company.com
tencent/hy3:xhigh__json_schema PASS 100% 397 in → 151 out (112 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling-small:high PASS 100% 394 in → 79 out (64 reasoning)
releaser@company.com
thinkingmachines/inkling-small:high__json_schema PASS 100% 394 in → 79 out (61 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling-small:low PASS 100% 402 in → 24 out (12 reasoning)
releaser@company.com
thinkingmachines/inkling-small:low__json_schema PASS 100% 404 in → 33 out (15 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling-small:max PASS 100% 402 in → 84 out (71 reasoning)
releaser@company.com
thinkingmachines/inkling-small:max__json_schema PASS 100% 398 in → 95 out (77 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling-small:medium PASS 100% 398 in → 22 out (10 reasoning)
releaser@company.com
thinkingmachines/inkling-small:medium__json_schema PASS 100% 392 in → 90 out (71 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling-small:none PASS 100% 388 in → 9 out (0 reasoning)
releaser@company.com
thinkingmachines/inkling-small:xhigh PASS 100% 396 in → 87 out (75 reasoning)
releaser@company.com
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 392 in → 91 out (73 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling:high PASS 100% 396 in → 99 out (86 reasoning)
releaser@company.com
thinkingmachines/inkling:high__json_schema PASS 100% 398 in → 97 out (79 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling:low PASS 100% 406 in → 39 out (26 reasoning)
releaser@company.com
thinkingmachines/inkling:max PASS 100% 396 in → 104 out (91 reasoning)
releaser@company.com
thinkingmachines/inkling:max__json_schema PASS 100% 394 in → 103 out (85 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling:medium PASS 100% 392 in → 110 out (98 reasoning)
releaser@company.com
thinkingmachines/inkling:none__json_schema PASS 100% 394 in → 20 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
thinkingmachines/inkling:xhigh PASS 100% 398 in → 63 out (49 reasoning)
releaser@company.com
thinkingmachines/inkling:xhigh__json_schema PASS 100% 392 in → 96 out (78 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
x-ai/grok-4.3:high PASS 100% 568 in → 124 out (118 reasoning)
releaser@company.com
x-ai/grok-4.3:high__json_schema PASS 100% 617 in → 281 out (271 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.3:low PASS 100% 572 in → 261 out (255 reasoning)
releaser@company.com
x-ai/grok-4.3:low__json_schema PASS 100% 621 in → 268 out (257 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
x-ai/grok-4.3:max PASS 100% 556 in → 258 out (252 reasoning)
releaser@company.com
x-ai/grok-4.3:max__json_schema PASS 100% 615 in → 210 out (200 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.3:medium PASS 100% 564 in → 131 out (125 reasoning)
releaser@company.com
x-ai/grok-4.3:medium__json_schema PASS 100% 617 in → 281 out (271 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.3:none PASS 100% 562 in → 6 out (0 reasoning)
releaser@company.com
x-ai/grok-4.3:none__json_schema PASS 100% 609 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
x-ai/grok-4.3:xhigh PASS 100% 566 in → 163 out (157 reasoning)
releaser@company.com
x-ai/grok-4.3:xhigh__json_schema PASS 100% 621 in → 330 out (320 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.5:high PASS 100% 602 in → 187 out (182 reasoning)
releaser@company.com
x-ai/grok-4.5:high__json_schema PASS 100% 657 in → 158 out (148 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.5:low PASS 100% 596 in → 119 out (114 reasoning)
releaser@company.com
x-ai/grok-4.5:low__json_schema PASS 100% 657 in → 107 out (97 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.5:max PASS 100% 592 in → 193 out (188 reasoning)
releaser@company.com
x-ai/grok-4.5:max__json_schema PASS 100% 655 in → 200 out (190 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.5:medium PASS 100% 584 in → 129 out (124 reasoning)
releaser@company.com
x-ai/grok-4.5:medium__json_schema PASS 100% 661 in → 258 out (248 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
x-ai/grok-4.5:xhigh PASS 100% 594 in → 183 out (178 reasoning)
releaser@company.com
x-ai/grok-4.5:xhigh__json_schema PASS 100% 653 in → 150 out (140 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email":"releaser@company.com"}
z-ai/glm-4.7-flash:high PASS 100% 400 in → 438 out (392 reasoning)
releaser@company.com
z-ai/glm-4.7-flash:high__json_schema PASS 100% 400 in → 474 out (401 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-4.7-flash:low PASS 100% 398 in → 411 out (403 reasoning)
releaser@company.com
z-ai/glm-4.7-flash:medium PASS 100% 402 in → 502 out (448 reasoning)
releaser@company.com
z-ai/glm-4.7-flash:none PASS 100% 396 in → 7 out (0 reasoning)
releaser@company.com
z-ai/glm-4.7-flash:none__json_schema PASS 100% 408 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-4.7-flash:xhigh PASS 100% 402 in → 406 out (378 reasoning)
releaser@company.com
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 406 in → 536 out (447 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-5.2:high PASS 100% 402 in → 264 out (206 reasoning)
releaser@company.com
z-ai/glm-5.2:high__json_schema PASS 100% 405 in → 181 out (168 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) {"email": "releaser@company.com"}
z-ai/glm-5.2:low PASS 100% 407 in → 236 out (0 reasoning)
releaser@company.com
z-ai/glm-5.2:low__json_schema PASS 100% 409 in → 262 out (226 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-5.2:medium PASS 100% 405 in → 171 out (163 reasoning)
releaser@company.com
z-ai/glm-5.2:medium__json_schema PASS 100% 407 in → 244 out (216 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-5.2:none PASS 100% 409 in → 7 out (0 reasoning)
releaser@company.com
z-ai/glm-5.2:none__json_schema PASS 100% 406 in → 14 out (0 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
z-ai/glm-5.2:xhigh PASS 100% 407 in → 288 out (280 reasoning)
releaser@company.com
z-ai/glm-5.2:xhigh__json_schema PASS 100% 409 in → 14 out (13 reasoning)
releaser@company.com
JSON Schema Structured Output
(raw) { "email": "releaser@company.com" }
anthropic/claude-fable-5:high FAIL 0% 524 in → 1 out (0 reasoning)
(empty output)
Failure: Expected 'releaser@company.com', got ''
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:low__json_schema FAIL 0% 480 in → 149 out (95 reasoning)
releaser@company.com>... Wait, I will just output the email. Wait, the email. Output ONLY the email, nothing else. Email: releaser@company.com>
JSON Schema Structured Output
(raw) {"email": "releaser@company.com>... Wait, I will just output the email. Wait, the email. Output ONLY the email, nothing else. Email: releaser@company.com>"}
Failure: Expected 'releaser@company.com', got 'releaser@company.com>... Wait, I will just output the email. Wait, the email. Output ONLY the email, nothing else. Email: releaser@company.com>'
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ee363895-2b3c-9fd2-8de0-19fdb92cbd8f","request_id":"ee363895-2b3c-9fd2-8de0-19fdb92cbd8f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4b01b188-8253-9e2c-9990-9795ee74b618","request_id":"4b01b188-8253-9e2c-9990-9795ee74b618"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7753369d-c664-9f3f-bb37-403e049ffeba","request_id":"7753369d-c664-9f3f-bb37-403e049ffeba"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4f88fbae-a1c5-9eb6-ae38-941f643c6067","request_id":"4f88fbae-a1c5-9eb6-ae38-941f643c6067"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cbaada86-303d-98df-b2a4-f058bd0914d1","request_id":"cbaada86-303d-98df-b2a4-f058bd0914d1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9697b906-cbb2-9bd4-af1e-97d2a97f99fb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-12d98d01-ad1b-980d-9a1a-352728bed497"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ea47c83e-800e-90f8-95f6-2e3635315694"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dd01dc00-854e-90dd-96c1-00135418342e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-91d66ace-169f-97bb-96c4-249dd180a5ff"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9a437e04-92a4-92fe-94e9-791624fea410"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ad1cb9d-8917-93b9-9bf6-9117c81627ea","request_id":"2ad1cb9d-8917-93b9-9bf6-9117c81627ea"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9fc20588-ae92-9199-8bd4-8515df13d2f5","request_id":"9fc20588-ae92-9199-8bd4-8515df13d2f5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-617e5d4a-3d91-9b49-9e14-631e013120a9","request_id":"617e5d4a-3d91-9b49-9e14-631e013120a9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f767bb36-288a-95b8-9876-f8fe807d1064","request_id":"f767bb36-288a-95b8-9876-f8fe807d1064"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aa84ba5d-ced6-9461-81db-896d19cdbb69","request_id":"aa84ba5d-ced6-9461-81db-896d19cdbb69"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3727b3a6-e367-9347-9c52-34fee1c2a0ef","request_id":"3727b3a6-e367-9347-9c52-34fee1c2a0ef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f493a12d-62d0-9242-b532-c37de92a5d38","request_id":"f493a12d-62d0-9242-b532-c37de92a5d38"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b377dd7a-440a-9c72-8bf5-103e6add79bc","request_id":"b377dd7a-440a-9c72-8bf5-103e6add79bc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-85e21a17-db40-92c8-893b-0a55ff6adf7b","request_id":"85e21a17-db40-92c8-893b-0a55ff6adf7b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3956455b-ca5c-9b12-b3b5-67df86cfe3f0","request_id":"3956455b-ca5c-9b12-b3b5-67df86cfe3f0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com{"email": "releaser@company.com"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:medium__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 392 in → 9 out (0 reasoning)
releaser@company.com<|end_message|>
Failure: Expected 'releaser@company.com', got 'releaser@company.com<|end_message|>'
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium__json_schema FAIL 0%
Invalid JSON. Output: releaser@company.com
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)