Identify most changed file from stat output
Tests ability to identify the most-changed file from stat output. Evaluates parsing git log --stat for change frequency.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'line1 line2 line3 ' > small.txt
  5. 05 printf 'a b ' > medium.txt
  6. 06 printf 'x y z w v u t s r ' > large.txt
  7. 07 git add small.txt medium.txt large.txt
  8. 08 git commit -m 'Initial commit with three files'
  9. 09 printf 'line1 line2 line3 line4 line5 ' > small.txt
  10. 10 printf 'a b c d e f g h i j k l m n o ' > medium.txt
  11. 11 printf 'x y z w v u t s r q p o n m l k j i h g f e d c b a ' > large.txt
  12. 12 git add small.txt medium.txt large.txt
  13. 13 git commit -m 'Update all files'
Prompt
In the most recent commit, which file had the most lines changed according to git log --stat? Output ONLY the filename, nothing else.
Expected
large.txt

Scoped model quality, cost, API time, and token usage for git_log_format/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 488 in → 6 out (0 reasoning)
large.txt
anthropic/claude-fable-5:high__json_schema PASS 100% 704 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-fable-5:low PASS 100% 487 in → 6 out (0 reasoning)
large.txt
anthropic/claude-fable-5:low__json_schema PASS 100% 693 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-fable-5:max PASS 100% 478 in → 88 out (51 reasoning)
large.txt
anthropic/claude-fable-5:max__json_schema PASS 100% 711 in → 108 out (44 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-fable-5:medium PASS 100% 503 in → 6 out (0 reasoning)
large.txt
anthropic/claude-fable-5:medium__json_schema PASS 100% 689 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-fable-5:xhigh PASS 100% 487 in → 6 out (0 reasoning)
large.txt
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 685 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-haiku-4.5:high PASS 100% 433 in → 241 out (198 reasoning)
large.txt
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 598 in → 236 out (189 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-haiku-4.5:low PASS 100% 436 in → 174 out (128 reasoning)
large.txt
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 612 in → 203 out (155 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-haiku-4.5:medium PASS 100% 454 in → 200 out (160 reasoning)
large.txt
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 591 in → 227 out (184 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-haiku-4.5:none PASS 100% 415 in → 6 out (0 reasoning)
large.txt
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 574 in → 11 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 445 in → 193 out (153 reasoning)
large.txt
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 601 in → 202 out (148 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-opus-4.6:high PASS 100% 404 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:high__json_schema PASS 100% 567 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.6:low PASS 100% 405 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:low__json_schema PASS 100% 570 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.6:max PASS 100% 426 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:max__json_schema PASS 100% 563 in → 99 out (60 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.6:medium PASS 100% 408 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 565 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.6:none PASS 100% 413 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:none__json_schema PASS 100% 573 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.6:xhigh PASS 100% 409 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 565 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:high PASS 100% 499 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:high__json_schema PASS 100% 693 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:low PASS 100% 500 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:low__json_schema PASS 100% 697 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:max PASS 100% 472 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:max__json_schema PASS 100% 693 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:medium PASS 100% 493 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 698 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:none PASS 100% 498 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:none__json_schema PASS 100% 700 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.7:xhigh PASS 100% 499 in → 8 out (0 reasoning)
large.txt
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 691 in → 16 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.8:high PASS 100% 487 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.8:high__json_schema PASS 100% 680 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.8:low PASS 100% 487 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.8:low__json_schema PASS 100% 681 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.8:max PASS 100% 478 in → 102 out (37 reasoning)
large.txt
anthropic/claude-opus-4.8:max__json_schema PASS 100% 682 in → 97 out (56 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-opus-4.8:medium PASS 100% 488 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 709 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.8:none PASS 100% 500 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.8:none__json_schema PASS 100% 700 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-opus-4.8:xhigh PASS 100% 497 in → 6 out (0 reasoning)
large.txt
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 689 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:high PASS 100% 415 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 573 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:low PASS 100% 404 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 568 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:max PASS 100% 410 in → 120 out (87 reasoning)
large.txt
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 569 in → 140 out (94 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:medium PASS 100% 421 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 564 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:none PASS 100% 409 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 571 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 404 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 569 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-5:high PASS 100% 504 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-5:high__json_schema PASS 100% 693 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-sonnet-5:low PASS 100% 475 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-5:low__json_schema PASS 100% 701 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-sonnet-5:medium PASS 100% 479 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 692 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
anthropic/claude-sonnet-5:none PASS 100% 472 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-5:none__json_schema PASS 100% 681 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
anthropic/claude-sonnet-5:xhigh PASS 100% 479 in → 6 out (0 reasoning)
large.txt
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 692 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
arcee-ai/trinity-large-thinking:high PASS 100% 341 in → 294 out (293 reasoning)
large.txt
arcee-ai/trinity-large-thinking:low PASS 100% 351 in → 265 out (260 reasoning)
large.txt
arcee-ai/trinity-large-thinking:medium PASS 100% 346 in → 226 out (221 reasoning)
large.txt
arcee-ai/trinity-large-thinking:xhigh PASS 100% 346 in → 419 out (419 reasoning)
large.txt
arcee-ai/trinity-mini:high PASS 100% 336 in → 163 out (167 reasoning)
large.txt
arcee-ai/trinity-mini:high__json_schema PASS 100% 351 in → 342 out (383 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
arcee-ai/trinity-mini:low PASS 100% 345 in → 246 out (251 reasoning)
large.txt
arcee-ai/trinity-mini:low__json_schema PASS 100% 347 in → 181 out (178 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
arcee-ai/trinity-mini:medium PASS 100% 345 in → 146 out (157 reasoning)
large.txt
arcee-ai/trinity-mini:medium__json_schema PASS 100% 343 in → 183 out (179 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
arcee-ai/trinity-mini:xhigh PASS 100% 338 in → 154 out (167 reasoning)
large.txt
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 348 in → 217 out (219 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename":"large.txt" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 353 in → 123 out (124 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 432 in → 55 out (48 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 346 in → 66 out (67 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 353 in → 114 out (110 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 437 in → 100 out (98 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 501 in → 48 out (42 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 352 in → 183 out (178 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 350 in → 87 out (76 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 350 in → 2 out (0 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 355 in → 12 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 432 in → 44 out (41 reasoning)
large.txt
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 432 in → 94 out (92 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-flash:high PASS 100% 346 in → 130 out (126 reasoning)
large.txt
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 361 in → 93 out (81 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
deepseek/deepseek-v4-flash:low PASS 100% 341 in → 67 out (62 reasoning)
large.txt
deepseek/deepseek-v4-flash:medium PASS 100% 346 in → 101 out (94 reasoning)
large.txt
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 444 in → 7 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-flash:none PASS 100% 366 in → 2 out (0 reasoning)
large.txt
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 425 in → 7 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 436 in → 180 out (176 reasoning)
large.txt
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 348 in → 75 out (62 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
deepseek/deepseek-v4-pro:high PASS 100% 362 in → 145 out (141 reasoning)
large.txt
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 540 in → 169 out (161 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
deepseek/deepseek-v4-pro:low PASS 100% 355 in → 3 out (0 reasoning)
large.txt
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 364 in → 130 out (120 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
deepseek/deepseek-v4-pro:medium PASS 100% 353 in → 102 out (98 reasoning)
large.txt
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 537 in → 91 out (82 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
deepseek/deepseek-v4-pro:none PASS 100% 364 in → 3 out (0 reasoning)
large.txt
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 535 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 439 in → 144 out (141 reasoning)
large.txt
google/gemini-3-flash-preview:high PASS 100% 441 in → 386 out (383 reasoning)
large.txt
google/gemini-3-flash-preview:high__json_schema PASS 100% 504 in → 277 out (270 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3-flash-preview:low PASS 100% 446 in → 299 out (296 reasoning)
large.txt
google/gemini-3-flash-preview:low__json_schema PASS 100% 475 in → 292 out (285 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3-flash-preview:medium PASS 100% 438 in → 249 out (246 reasoning)
large.txt
google/gemini-3-flash-preview:medium__json_schema PASS 100% 436 in → 348 out (340 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3-flash-preview:xhigh PASS 100% 438 in → 257 out (254 reasoning)
large.txt
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 509 in → 296 out (289 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 445 in → 554 out (551 reasoning)
large.txt
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 491 in → 250 out (243 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 447 in → 119 out (116 reasoning)
large.txt
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 451 in → 143 out (130 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 441 in → 180 out (177 reasoning)
large.txt
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 482 in → 269 out (261 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 446 in → 236 out (233 reasoning)
large.txt
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 431 in → 382 out (370 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemini-3.1-pro-preview:high PASS 100% 427 in → 284 out (281 reasoning)
large.txt
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 495 in → 469 out (462 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.1-pro-preview:low PASS 100% 448 in → 123 out (120 reasoning)
large.txt
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 500 in → 125 out (117 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3.1-pro-preview:medium PASS 100% 436 in → 260 out (257 reasoning)
large.txt
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 490 in → 242 out (235 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 445 in → 336 out (333 reasoning)
large.txt
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 491 in → 353 out (346 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.5-flash-lite:high PASS 100% 441 in → 271 out (268 reasoning)
large.txt
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 457 in → 335 out (327 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3.5-flash-lite:low PASS 100% 449 in → 3 out (0 reasoning)
large.txt
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 499 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3.5-flash-lite:max PASS 100% 438 in → 296 out (293 reasoning)
large.txt
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 438 in → 411 out (404 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.5-flash-lite:medium PASS 100% 429 in → 161 out (158 reasoning)
large.txt
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 452 in → 380 out (373 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 442 in → 311 out (308 reasoning)
large.txt
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 497 in → 401 out (393 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
google/gemini-3.5-flash:high PASS 100% 441 in → 501 out (498 reasoning)
large.txt
google/gemini-3.5-flash:high__json_schema PASS 100% 501 in → 443 out (430 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemini-3.5-flash:low PASS 100% 443 in → 168 out (165 reasoning)
large.txt
google/gemini-3.5-flash:low__json_schema PASS 100% 486 in → 160 out (153 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.5-flash:medium PASS 100% 437 in → 302 out (299 reasoning)
large.txt
google/gemini-3.5-flash:medium__json_schema PASS 100% 489 in → 318 out (311 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.5-flash:xhigh PASS 100% 447 in → 590 out (587 reasoning)
large.txt
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 489 in → 244 out (237 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.6-flash:high PASS 100% 444 in → 361 out (358 reasoning)
large.txt
google/gemini-3.6-flash:high__json_schema PASS 100% 508 in → 273 out (260 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemini-3.6-flash:low PASS 100% 442 in → 133 out (130 reasoning)
large.txt
google/gemini-3.6-flash:low__json_schema PASS 100% 493 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemini-3.6-flash:max PASS 100% 456 in → 320 out (317 reasoning)
large.txt
google/gemini-3.6-flash:max__json_schema PASS 100% 443 in → 416 out (409 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.6-flash:medium PASS 100% 444 in → 212 out (209 reasoning)
large.txt
google/gemini-3.6-flash:medium__json_schema PASS 100% 432 in → 342 out (335 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemini-3.6-flash:xhigh PASS 100% 446 in → 248 out (245 reasoning)
large.txt
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 448 in → 325 out (318 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemma-4-26b-a4b-it:high PASS 100% 450 in → 254 out (183 reasoning)
large.txt
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 458 in → 311 out (205 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-26b-a4b-it:low PASS 100% 450 in → 344 out (228 reasoning)
large.txt
google/gemma-4-26b-a4b-it:medium PASS 100% 449 in → 285 out (187 reasoning)
large.txt
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 449 in → 311 out (198 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-26b-a4b-it:none PASS 100% 456 in → 4 out (0 reasoning)
large.txt
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 459 in → 14 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 465 in → 328 out (208 reasoning)
large.txt
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 463 in → 358 out (231 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-31b-it:high PASS 100% 459 in → 230 out (136 reasoning)
large.txt
google/gemma-4-31b-it:high__json_schema PASS 100% 459 in → 189 out (135 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-31b-it:low PASS 100% 457 in → 210 out (157 reasoning)
large.txt
google/gemma-4-31b-it:low__json_schema PASS 100% 451 in → 219 out (156 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-31b-it:medium PASS 100% 445 in → 260 out (199 reasoning)
large.txt
google/gemma-4-31b-it:medium__json_schema PASS 100% 460 in → 257 out (248 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
google/gemma-4-31b-it:none PASS 100% 450 in → 4 out (0 reasoning)
large.txt
google/gemma-4-31b-it:none__json_schema PASS 100% 453 in → 11 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
google/gemma-4-31b-it:xhigh PASS 100% 454 in → 278 out (1 reasoning)
large.txt
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 443 in → 218 out (133 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
ibm-granite/granite-4.1-8b PASS 100% 380 in → 3 out
large.txt
ibm-granite/granite-4.1-8b__json_schema PASS 100% 374 in → 10 out
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
inclusionai/ling-2.6-flash PASS 100% 464 in → 3 out
large.txt
inclusionai/ling-2.6-flash__json_schema PASS 100% 463 in → 13 out
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
liquid/lfm-2-24b-a2b PASS 100% 381 in → 3 out
large.txt
minimax/minimax-m2.5:high PASS 100% 379 in → 172 out (168 reasoning)
large.txt
minimax/minimax-m2.5:high__json_schema PASS 100% 382 in → 271 out (282 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m2.5:low PASS 100% 374 in → 249 out (271 reasoning)
large.txt
minimax/minimax-m2.5:low__json_schema PASS 100% 384 in → 240 out (259 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m2.5:medium PASS 100% 383 in → 154 out (152 reasoning)
large.txt
minimax/minimax-m2.5:medium__json_schema PASS 100% 408 in → 100 out (87 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m2.5:xhigh PASS 100% 382 in → 161 out (163 reasoning)
large.txt
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 389 in → 258 out (248 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m2.7:high PASS 100% 384 in → 175 out (172 reasoning)
large.txt
minimax/minimax-m2.7:high__json_schema PASS 100% 530 in → 482 out (474 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
minimax/minimax-m2.7:low PASS 100% 383 in → 120 out (127 reasoning)
large.txt
minimax/minimax-m2.7:low__json_schema PASS 100% 532 in → 258 out (250 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
minimax/minimax-m2.7:medium PASS 100% 390 in → 79 out (76 reasoning)
large.txt
minimax/minimax-m2.7:medium__json_schema PASS 100% 504 in → 273 out (265 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
minimax/minimax-m2.7:xhigh PASS 100% 362 in → 135 out (145 reasoning)
large.txt
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 513 in → 154 out (142 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m3:high PASS 100% 530 in → 64 out (62 reasoning)
large.txt
minimax/minimax-m3:high__json_schema PASS 100% 497 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
minimax/minimax-m3:low PASS 100% 531 in → 81 out (80 reasoning)
large.txt
minimax/minimax-m3:low__json_schema PASS 100% 512 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m3:medium PASS 100% 511 in → 61 out (59 reasoning)
large.txt
minimax/minimax-m3:medium__json_schema PASS 100% 508 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
minimax/minimax-m3:xhigh PASS 100% 535 in → 71 out (70 reasoning)
large.txt
minimax/minimax-m3:xhigh__json_schema PASS 100% 488 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/devstral-2512 PASS 100% 440 in → 3 out
large.txt
mistralai/devstral-2512__json_schema PASS 100% 439 in → 8 out
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/mistral-medium-3-5:high PASS 100% 447 in → 260 out (244 reasoning)
large.txt
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 443 in → 216 out (200 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/mistral-medium-3-5:low PASS 100% 453 in → 398 out (382 reasoning)
large.txt
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 446 in → 220 out (176 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/mistral-medium-3-5:medium PASS 100% 444 in → 367 out (349 reasoning)
large.txt
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 441 in → 266 out (235 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/mistral-medium-3-5:none PASS 100% 461 in → 3 out (0 reasoning)
large.txt
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 439 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 466 in → 164 out (153 reasoning)
large.txt
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 434 in → 240 out (226 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
moonshotai/kimi-k2.6:high PASS 100% 339 in → 151 out (147 reasoning)
large.txt
moonshotai/kimi-k2.6:high__json_schema PASS 100% 535 in → 194 out (186 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
moonshotai/kimi-k2.6:low PASS 100% 350 in → 108 out (104 reasoning)
large.txt
moonshotai/kimi-k2.6:low__json_schema PASS 100% 355 in → 284 out (130 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
moonshotai/kimi-k2.6:medium PASS 100% 340 in → 203 out (200 reasoning)
large.txt
moonshotai/kimi-k2.6:xhigh PASS 100% 343 in → 263 out (283 reasoning)
large.txt
moonshotai/kimi-k2.7-code:high PASS 100% 340 in → 82 out (87 reasoning)
large.txt
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 347 in → 105 out (98 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
moonshotai/kimi-k2.7-code:low PASS 100% 351 in → 61 out (57 reasoning)
large.txt
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 344 in → 90 out (84 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
moonshotai/kimi-k2.7-code:medium PASS 100% 349 in → 98 out (94 reasoning)
large.txt
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 344 in → 50 out (35 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 354 in → 98 out (94 reasoning)
large.txt
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 348 in → 71 out (59 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
moonshotai/kimi-k3:high PASS 100% 432 in → 112 out (95 reasoning)
large.txt
moonshotai/kimi-k3:high__json_schema PASS 100% 517 in → 117 out (96 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
moonshotai/kimi-k3:low PASS 100% 430 in → 76 out (59 reasoning)
large.txt
moonshotai/kimi-k3:low__json_schema PASS 100% 422 in → 72 out (55 reasoning)
large.txt
moonshotai/kimi-k3:max PASS 100% 437 in → 74 out (57 reasoning)
large.txt
moonshotai/kimi-k3:max__json_schema PASS 100% 428 in → 98 out (81 reasoning)
large.txt
moonshotai/kimi-k3:medium PASS 100% 430 in → 105 out (88 reasoning)
large.txt
moonshotai/kimi-k3:medium__json_schema PASS 100% 435 in → 77 out (60 reasoning)
large.txt
moonshotai/kimi-k3:xhigh PASS 100% 423 in → 72 out (55 reasoning)
large.txt
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 536 in → 217 out (196 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 464 in → 80 out (70 reasoning)
large.txt
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 455 in → 141 out (129 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 454 in → 289 out (272 reasoning)
large.txt
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 469 in → 160 out (151 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 448 in → 94 out (88 reasoning)
large.txt
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 460 in → 145 out (133 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 462 in → 3 out (0 reasoning)
large.txt
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 460 in → 12 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 455 in → 139 out (128 reasoning)
large.txt
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 457 in → 154 out (152 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 461 in → 74 out (66 reasoning)
large.txt
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 463 in → 157 out (118 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 460 in → 109 out (94 reasoning)
large.txt
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 462 in → 82 out (66 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 458 in → 84 out (74 reasoning)
large.txt
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 461 in → 94 out (78 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 470 in → 3 out (0 reasoning)
large.txt
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 451 in → 11 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 458 in → 108 out (96 reasoning)
large.txt
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 454 in → 91 out (76 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
openai/gpt-5.4-mini:high PASS 100% 349 in → 59 out (51 reasoning)
large.txt
openai/gpt-5.4-mini:high__json_schema PASS 100% 389 in → 244 out (229 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-mini:low PASS 100% 342 in → 41 out (33 reasoning)
large.txt
openai/gpt-5.4-mini:low__json_schema PASS 100% 383 in → 44 out (29 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-mini:medium PASS 100% 345 in → 38 out (30 reasoning)
large.txt
openai/gpt-5.4-mini:medium__json_schema PASS 100% 382 in → 134 out (119 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-mini:none PASS 100% 346 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.4-mini:none__json_schema PASS 100% 377 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-mini:xhigh PASS 100% 347 in → 74 out (66 reasoning)
large.txt
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 393 in → 458 out (443 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-nano:high PASS 100% 358 in → 51 out (43 reasoning)
large.txt
openai/gpt-5.4-nano:high__json_schema PASS 100% 384 in → 76 out (61 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-nano:low PASS 100% 347 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.4-nano:low__json_schema PASS 100% 388 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-nano:medium PASS 100% 364 in → 77 out (69 reasoning)
large.txt
openai/gpt-5.4-nano:medium__json_schema PASS 100% 375 in → 52 out (37 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-nano:none PASS 100% 350 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.4-nano:none__json_schema PASS 100% 377 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4-nano:xhigh PASS 100% 353 in → 84 out (76 reasoning)
large.txt
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 373 in → 134 out (119 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4:high PASS 100% 342 in → 58 out (50 reasoning)
large.txt
openai/gpt-5.4:high__json_schema PASS 100% 386 in → 73 out (58 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4:low PASS 100% 360 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.4:low__json_schema PASS 100% 385 in → 37 out (22 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4:medium PASS 100% 358 in → 35 out (27 reasoning)
large.txt
openai/gpt-5.4:medium__json_schema PASS 100% 390 in → 52 out (37 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4:none PASS 100% 357 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.4:none__json_schema PASS 100% 377 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.4:xhigh PASS 100% 351 in → 76 out (68 reasoning)
large.txt
openai/gpt-5.4:xhigh__json_schema PASS 100% 380 in → 80 out (65 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.5:high PASS 100% 345 in → 41 out (33 reasoning)
large.txt
openai/gpt-5.5:high__json_schema PASS 100% 375 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.5:low PASS 100% 346 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.5:low__json_schema PASS 100% 375 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.5:medium PASS 100% 361 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.5:medium__json_schema PASS 100% 371 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.5:none PASS 100% 355 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.5:none__json_schema PASS 100% 381 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.5:xhigh PASS 100% 349 in → 59 out (51 reasoning)
large.txt
openai/gpt-5.5:xhigh__json_schema PASS 100% 380 in → 177 out (162 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:high PASS 100% 344 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-luna:high__json_schema PASS 100% 386 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:low PASS 100% 346 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-luna:low__json_schema PASS 100% 371 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:max PASS 100% 352 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-luna:max__json_schema PASS 100% 375 in → 51 out (36 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:medium PASS 100% 361 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-luna:medium__json_schema PASS 100% 388 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:none PASS 100% 353 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-luna:none__json_schema PASS 100% 386 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-luna:xhigh PASS 100% 355 in → 40 out (32 reasoning)
large.txt
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 382 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:high PASS 100% 353 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:high__json_schema PASS 100% 378 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:low PASS 100% 350 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:low__json_schema PASS 100% 373 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:max PASS 100% 343 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:max__json_schema PASS 100% 388 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:medium PASS 100% 351 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:medium__json_schema PASS 100% 388 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:none PASS 100% 356 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:none__json_schema PASS 100% 370 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-sol:xhigh PASS 100% 361 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 381 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:high PASS 100% 350 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:high__json_schema PASS 100% 377 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:low PASS 100% 342 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:low__json_schema PASS 100% 378 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:max PASS 100% 365 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:max__json_schema PASS 100% 379 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:medium PASS 100% 349 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:medium__json_schema PASS 100% 389 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:none PASS 100% 345 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:none__json_schema PASS 100% 384 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-5.6-terra:xhigh PASS 100% 355 in → 6 out (0 reasoning)
large.txt
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 386 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-120b:high PASS 100% 411 in → 212 out (220 reasoning)
large.txt
openai/gpt-oss-120b:high__json_schema PASS 100% 412 in → 220 out (228 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-120b:low PASS 100% 402 in → 54 out (38 reasoning)
large.txt
openai/gpt-oss-120b:low__json_schema PASS 100% 405 in → 63 out (44 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-120b:medium PASS 100% 415 in → 123 out (115 reasoning)
large.txt
openai/gpt-oss-120b:medium__json_schema PASS 100% 400 in → 122 out (109 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-120b:xhigh PASS 100% 389 in → 166 out (166 reasoning)
large.txt
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 414 in → 204 out (188 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-20b:high PASS 100% 406 in → 149 out (139 reasoning)
large.txt
openai/gpt-oss-20b:high__json_schema PASS 100% 405 in → 263 out (256 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
openai/gpt-oss-20b:low PASS 100% 387 in → 52 out (35 reasoning)
large.txt
openai/gpt-oss-20b:low__json_schema PASS 100% 449 in → 65 out (46 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-20b:medium PASS 100% 421 in → 136 out (130 reasoning)
large.txt
openai/gpt-oss-20b:medium__json_schema PASS 100% 391 in → 137 out (129 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
openai/gpt-oss-20b:xhigh PASS 100% 409 in → 176 out (184 reasoning)
large.txt
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 391 in → 243 out (226 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
poolside/laguna-m.1:high PASS 100% 439 in → 335 out (330 reasoning)
large.txt
poolside/laguna-m.1:high__json_schema PASS 100% 436 in → 194 out (185 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-m.1:low PASS 100% 441 in → 212 out (207 reasoning)
large.txt
poolside/laguna-m.1:low__json_schema PASS 100% 452 in → 299 out (290 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-m.1:medium PASS 100% 437 in → 205 out (200 reasoning)
large.txt
poolside/laguna-m.1:medium__json_schema PASS 100% 431 in → 181 out (168 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-m.1:none PASS 100% 442 in → 4 out (0 reasoning)
large.txt
poolside/laguna-m.1:none__json_schema PASS 100% 439 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-m.1:xhigh PASS 100% 441 in → 336 out (331 reasoning)
large.txt
poolside/laguna-m.1:xhigh__json_schema PASS 100% 434 in → 215 out (201 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-xs-2.1:high PASS 100% 433 in → 367 out (364 reasoning)
large.txt
poolside/laguna-xs-2.1:high__json_schema PASS 100% 445 in → 201 out (186 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-xs-2.1:low PASS 100% 450 in → 389 out (386 reasoning)
large.txt
poolside/laguna-xs-2.1:low__json_schema PASS 100% 440 in → 325 out (315 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-xs-2.1:medium PASS 100% 431 in → 275 out (272 reasoning)
large.txt
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 457 in → 358 out (348 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-xs-2.1:none PASS 100% 448 in → 2 out (0 reasoning)
large.txt
poolside/laguna-xs-2.1:none__json_schema PASS 100% 448 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-xs-2.1:xhigh PASS 100% 447 in → 211 out (208 reasoning)
large.txt
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 441 in → 150 out (135 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-xs.2:high PASS 100% 484 in → 233 out (229 reasoning)
large.txt
poolside/laguna-xs.2:high__json_schema PASS 100% 487 in → 249 out (241 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-xs.2:low PASS 100% 471 in → 248 out (244 reasoning)
large.txt
poolside/laguna-xs.2:low__json_schema PASS 100% 474 in → 243 out (230 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
poolside/laguna-xs.2:medium PASS 100% 473 in → 127 out (122 reasoning)
large.txt
poolside/laguna-xs.2:medium__json_schema PASS 100% 483 in → 142 out (134 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
poolside/laguna-xs.2:none PASS 100% 473 in → 4 out (0 reasoning)
large.txt
poolside/laguna-xs.2:none__json_schema PASS 100% 477 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
poolside/laguna-xs.2:xhigh PASS 100% 477 in → 221 out (217 reasoning)
large.txt
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 473 in → 251 out (243 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
qwen/qwen3.6-27b:high PASS 100% 437 in → 382 out (325 reasoning)
large.txt
qwen/qwen3.6-27b:high__json_schema PASS 100% 429 in → 406 out (370 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-27b:low PASS 100% 441 in → 211 out (185 reasoning)
large.txt
qwen/qwen3.6-27b:medium PASS 100% 445 in → 222 out (214 reasoning)
large.txt
qwen/qwen3.6-27b:medium__json_schema PASS 100% 437 in → 339 out (249 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-27b:none PASS 100% 443 in → 3 out (0 reasoning)
large.txt
qwen/qwen3.6-27b:none__json_schema PASS 100% 440 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-27b:xhigh PASS 100% 442 in → 291 out (262 reasoning)
large.txt
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 427 in → 283 out (266 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-35b-a3b:high PASS 100% 435 in → 305 out (276 reasoning)
large.txt
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 432 in → 593 out (485 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
qwen/qwen3.6-35b-a3b:low PASS 100% 435 in → 183 out (171 reasoning)
large.txt
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 445 in → 329 out (279 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 441 in → 211 out (197 reasoning)
large.txt
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 458 in → 240 out (199 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-35b-a3b:none PASS 100% 444 in → 3 out (0 reasoning)
large.txt
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 451 in → 13 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 441 in → 259 out (252 reasoning)
large.txt
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 447 in → 201 out (154 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
qwen/qwen3.6-flash:high PASS 100% 437 in → 225 out (218 reasoning)
large.txt
qwen/qwen3.6-flash:low PASS 100% 444 in → 337 out (330 reasoning)
large.txt
qwen/qwen3.6-flash:medium PASS 100% 440 in → 365 out (359 reasoning)
large.txt
qwen/qwen3.6-flash:none PASS 100% 444 in → 2 out (0 reasoning)
large.txt
qwen/qwen3.6-flash:xhigh PASS 100% 445 in → 265 out (258 reasoning)
large.txt
qwen/qwen3.7-flash:high PASS 100% 439 in → 292 out (285 reasoning)
large.txt
qwen/qwen3.7-flash:low PASS 100% 426 in → 239 out (233 reasoning)
large.txt
qwen/qwen3.7-flash:max PASS 100% 450 in → 257 out (251 reasoning)
large.txt
qwen/qwen3.7-flash:medium PASS 100% 438 in → 237 out (230 reasoning)
large.txt
qwen/qwen3.7-flash:none PASS 100% 444 in → 2 out (0 reasoning)
large.txt
qwen/qwen3.7-flash:xhigh PASS 100% 435 in → 306 out (300 reasoning)
large.txt
qwen/qwen3.7-max:high PASS 100% 429 in → 177 out (170 reasoning)
large.txt
qwen/qwen3.7-max:low PASS 100% 443 in → 183 out (176 reasoning)
large.txt
qwen/qwen3.7-max:medium PASS 100% 442 in → 133 out (126 reasoning)
large.txt
qwen/qwen3.7-max:none PASS 100% 445 in → 2 out (0 reasoning)
large.txt
qwen/qwen3.7-max:xhigh PASS 100% 445 in → 161 out (154 reasoning)
large.txt
qwen/qwen3.7-plus:high PASS 100% 443 in → 474 out (467 reasoning)
large.txt
qwen/qwen3.7-plus:low PASS 100% 439 in → 148 out (141 reasoning)
large.txt
qwen/qwen3.7-plus:medium PASS 100% 438 in → 171 out (164 reasoning)
large.txt
qwen/qwen3.7-plus:none PASS 100% 444 in → 2 out (0 reasoning)
large.txt
qwen/qwen3.7-plus:xhigh PASS 100% 425 in → 178 out (171 reasoning)
large.txt
tencent/hy3:high PASS 100% 365 in → 127 out (123 reasoning)
large.txt
tencent/hy3:high__json_schema PASS 100% 355 in → 199 out (166 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
tencent/hy3:low PASS 100% 354 in → 150 out (146 reasoning)
large.txt
tencent/hy3:low__json_schema PASS 100% 346 in → 153 out (144 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
tencent/hy3:medium PASS 100% 344 in → 162 out (160 reasoning)
large.txt
tencent/hy3:medium__json_schema PASS 100% 362 in → 148 out (139 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
tencent/hy3:none PASS 100% 374 in → 3 out (0 reasoning)
large.txt
tencent/hy3:none__json_schema PASS 100% 351 in → 10 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
tencent/hy3:xhigh PASS 100% 353 in → 118 out (114 reasoning)
large.txt
tencent/hy3:xhigh__json_schema PASS 100% 363 in → 167 out (161 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling-small:high PASS 100% 356 in → 90 out (86 reasoning)
large.txt
thinkingmachines/inkling-small:high__json_schema PASS 100% 365 in → 82 out (67 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling-small:low PASS 100% 345 in → 40 out (32 reasoning)
large.txt
thinkingmachines/inkling-small:low__json_schema PASS 100% 364 in → 49 out (34 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling-small:max PASS 100% 366 in → 49 out (36 reasoning)
large.txt
thinkingmachines/inkling-small:max__json_schema PASS 100% 362 in → 53 out (37 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling-small:medium PASS 100% 357 in → 75 out (70 reasoning)
large.txt
thinkingmachines/inkling-small:medium__json_schema PASS 100% 352 in → 61 out (46 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling-small:none PASS 100% 362 in → 5 out (0 reasoning)
large.txt
thinkingmachines/inkling-small:xhigh PASS 100% 355 in → 72 out (65 reasoning)
large.txt
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 357 in → 53 out (38 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling:high PASS 100% 357 in → 71 out (62 reasoning)
large.txt
thinkingmachines/inkling:high__json_schema PASS 100% 358 in → 79 out (65 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling:low PASS 100% 361 in → 34 out (25 reasoning)
large.txt
thinkingmachines/inkling:low__json_schema PASS 100% 358 in → 38 out (24 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling:max PASS 100% 366 in → 58 out (48 reasoning)
large.txt
thinkingmachines/inkling:max__json_schema PASS 100% 372 in → 74 out (59 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
thinkingmachines/inkling:medium PASS 100% 362 in → 52 out (44 reasoning)
large.txt
thinkingmachines/inkling:medium__json_schema PASS 100% 352 in → 54 out (42 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
thinkingmachines/inkling:none PASS 100% 370 in → 5 out (0 reasoning)
large.txt
thinkingmachines/inkling:xhigh PASS 100% 360 in → 103 out (94 reasoning)
large.txt
thinkingmachines/inkling:xhigh__json_schema PASS 100% 363 in → 115 out (100 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
x-ai/grok-4.3:high PASS 100% 516 in → 223 out (221 reasoning)
large.txt
x-ai/grok-4.3:high__json_schema PASS 100% 563 in → 269 out (263 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.3:low PASS 100% 537 in → 184 out (182 reasoning)
large.txt
x-ai/grok-4.3:low__json_schema PASS 100% 579 in → 348 out (342 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.3:max PASS 100% 510 in → 200 out (198 reasoning)
large.txt
x-ai/grok-4.3:max__json_schema PASS 100% 586 in → 354 out (348 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.3:medium PASS 100% 536 in → 257 out (255 reasoning)
large.txt
x-ai/grok-4.3:medium__json_schema PASS 100% 580 in → 287 out (281 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.3:none PASS 100% 513 in → 2 out (0 reasoning)
large.txt
x-ai/grok-4.3:none__json_schema PASS 100% 584 in → 6 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.3:xhigh PASS 100% 519 in → 187 out (185 reasoning)
large.txt
x-ai/grok-4.3:xhigh__json_schema PASS 100% 584 in → 325 out (319 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.5:high PASS 100% 542 in → 196 out (194 reasoning)
large.txt
x-ai/grok-4.5:high__json_schema PASS 100% 624 in → 291 out (284 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.5:low PASS 100% 552 in → 150 out (148 reasoning)
large.txt
x-ai/grok-4.5:low__json_schema PASS 100% 634 in → 114 out (107 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.5:max PASS 100% 541 in → 196 out (194 reasoning)
large.txt
x-ai/grok-4.5:max__json_schema PASS 100% 613 in → 270 out (263 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
x-ai/grok-4.5:medium PASS 100% 546 in → 164 out (162 reasoning)
large.txt
x-ai/grok-4.5:medium__json_schema PASS 100% 622 in → 342 out (335 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
x-ai/grok-4.5:xhigh PASS 100% 556 in → 164 out (162 reasoning)
large.txt
x-ai/grok-4.5:xhigh__json_schema PASS 100% 613 in → 211 out (204 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename":"large.txt"}
z-ai/glm-4.7-flash:high PASS 100% 366 in → 611 out (512 reasoning)
large.txt
z-ai/glm-4.7-flash:high__json_schema PASS 100% 368 in → 414 out (356 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
z-ai/glm-4.7-flash:low PASS 100% 360 in → 360 out (326 reasoning)
large.txt
z-ai/glm-4.7-flash:medium PASS 100% 357 in → 539 out (473 reasoning)
large.txt
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 363 in → 382 out (344 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
z-ai/glm-4.7-flash:none PASS 100% 368 in → 3 out (0 reasoning)
large.txt
z-ai/glm-4.7-flash:none__json_schema PASS 100% 353 in → 8 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
z-ai/glm-4.7-flash:xhigh PASS 100% 362 in → 460 out (427 reasoning)
large.txt
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 360 in → 335 out (293 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
z-ai/glm-5.2:high PASS 100% 368 in → 198 out (194 reasoning)
large.txt
z-ai/glm-5.2:high__json_schema PASS 100% 365 in → 273 out (227 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt"}
z-ai/glm-5.2:low PASS 100% 383 in → 135 out (139 reasoning)
large.txt
z-ai/glm-5.2:low__json_schema PASS 100% 361 in → 225 out (175 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
z-ai/glm-5.2:medium PASS 100% 363 in → 127 out (123 reasoning)
large.txt
z-ai/glm-5.2:medium__json_schema PASS 100% 363 in → 111 out (109 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
z-ai/glm-5.2:none PASS 100% 372 in → 3 out (0 reasoning)
large.txt
z-ai/glm-5.2:none__json_schema PASS 100% 374 in → 9 out (0 reasoning)
large.txt
JSON Schema Structured Output
(raw) { "filename": "large.txt" }
z-ai/glm-5.2:xhigh PASS 100% 383 in → 245 out (241 reasoning)
large.txt
z-ai/glm-5.2:xhigh__json_schema PASS 100% 371 in → 8 out (7 reasoning)
large.txt
JSON Schema Structured Output
(raw) {"filename": "large.txt"}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: large.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: {
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 9782 column 1 (char 48904)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 9782 column 1 (char 48904)
Invalid JSON. Output: large.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: large.txt
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6164dee1-27fa-9506-9e23-0468a1f5bd63","request_id":"6164dee1-27fa-9506-9e23-0468a1f5bd63"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-495ea414-e2a8-90aa-87c2-98a03f866874","request_id":"495ea414-e2a8-90aa-87c2-98a03f866874"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b648f2d5-bc77-9c6e-b352-bababc013f49","request_id":"b648f2d5-bc77-9c6e-b352-bababc013f49"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c16b11a9-4c3d-9a69-a4f6-340625e6969d","request_id":"c16b11a9-4c3d-9a69-a4f6-340625e6969d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-980ac7a9-ee05-9865-afdd-d82d0afbb8a4","request_id":"980ac7a9-ee05-9865-afdd-d82d0afbb8a4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-78d31fce-be10-94ee-9555-4ec9abbb3623"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-d9298982-d843-944a-ad70-79fabe592523"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-04864f52-2c4d-994a-8d4a-3e117a9dba64"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-60fce12d-658c-9ce3-9023-b1f63f66b076"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9bbf3fec-0f3e-967a-a52c-cef0f795efb6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a3fdd2ef-1d9d-9f49-a39a-5e804ad26c35"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-921b6e92-81d9-92b9-a631-3589797273f7","request_id":"921b6e92-81d9-92b9-a631-3589797273f7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6e98afaf-92dc-91e3-ad48-e3394ef4a640","request_id":"6e98afaf-92dc-91e3-ad48-e3394ef4a640"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f8014489-14c8-9eab-912b-7d7b0e18e7e3","request_id":"f8014489-14c8-9eab-912b-7d7b0e18e7e3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e59a39b0-5368-924b-b542-2779f8fbe4fe","request_id":"e59a39b0-5368-924b-b542-2779f8fbe4fe"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5741c956-7768-9e73-b265-4aba26ea69f8","request_id":"5741c956-7768-9e73-b265-4aba26ea69f8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-60fbdafe-7ce2-9319-8ca0-28eba8139452","request_id":"60fbdafe-7ce2-9319-8ca0-28eba8139452"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-69c2aa2a-278b-9ea5-9aef-266b1fe79f16","request_id":"69c2aa2a-278b-9ea5-9aef-266b1fe79f16"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a524e8ce-1f24-9415-95cc-d51e376fc523","request_id":"a524e8ce-1f24-9415-95cc-d51e376fc523"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-00611a71-6c3f-9107-84d5-6cf4a5961b15","request_id":"00611a71-6c3f-9107-84d5-6cf4a5961b15"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f3b2db03-16e1-922d-8fa9-6be03bddbbff","request_id":"f3b2db03-16e1-922d-8fa9-6be03bddbbff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: large.txt{ "filename": "large.txt" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: large.txt{"filename": "large.txt"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)