Identify specific commit from oneline output
Tests ability to identify a specific commit from oneline output. Evaluates parsing compact log formats.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1' > app.py
  5. 05 git add app.py
  6. 06 git commit -m 'Initial application setup'
  7. 07 echo 'v2' > app.py
  8. 08 git add app.py
  9. 09 git commit -m 'Add error handling'
  10. 10 echo 'v3' > app.py
  11. 11 git add app.py
  12. 12 git commit -m 'Fix null pointer bug'
  13. 13 echo 'v4' > app.py
  14. 14 git add app.py
  15. 15 git commit -m 'Refactor module structure'
  16. 16 echo 'v5' > app.py
  17. 17 git add app.py
  18. 18 git commit -m 'Add unit tests'
Prompt
Using git log --oneline, what is the short hash of the commit with message 'Fix null pointer bug'? Output ONLY the short hash, nothing else.
Expected

Scoped model quality, cost, API time, and token usage for git_log_format/f007.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 975 in → 7 out (0 reasoning)
b421bcc
anthropic/claude-fable-5:high__json_schema PASS 100% 1,175 in → 14 out (0 reasoning)
ed72261
JSON Schema Structured Output
(raw) {"hash": "ed72261"}
anthropic/claude-fable-5:low PASS 100% 955 in → 5 out (0 reasoning)
a300530
anthropic/claude-fable-5:low__json_schema PASS 100% 1,183 in → 15 out (0 reasoning)
f698b3e
JSON Schema Structured Output
(raw) {"hash": "f698b3e"}
anthropic/claude-fable-5:max__json_schema PASS 100% 1,178 in → 41 out (24 reasoning)
6535555
JSON Schema Structured Output
(raw) {"hash": "6535555"}
anthropic/claude-fable-5:medium PASS 100% 999 in → 8 out (0 reasoning)
8bbdd71
anthropic/claude-fable-5:medium__json_schema PASS 100% 1,193 in → 15 out (0 reasoning)
cf544d0
JSON Schema Structured Output
(raw) {"hash": "cf544d0"}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 1,177 in → 15 out (0 reasoning)
58e39c9
JSON Schema Structured Output
(raw) {"hash":"58e39c9"}
anthropic/claude-haiku-4.5:high PASS 100% 851 in → 246 out (190 reasoning)
397b684
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 1,014 in → 349 out (331 reasoning)
a69acc6
JSON Schema Structured Output
(raw) {"hash": "a69acc6"}
anthropic/claude-haiku-4.5:low PASS 100% 844 in → 135 out (115 reasoning)
75111d2
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 1,003 in → 297 out (247 reasoning)
a90d34f
JSON Schema Structured Output
(raw) {"hash": "a90d34f"}
anthropic/claude-haiku-4.5:medium PASS 100% 833 in → 127 out (108 reasoning)
133d756
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 1,000 in → 363 out (328 reasoning)
c03c538
JSON Schema Structured Output
(raw) {"hash":"c03c538"}
anthropic/claude-haiku-4.5:none PASS 100% 794 in → 8 out (0 reasoning)
2262d17
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 974 in → 14 out (0 reasoning)
bfbd5e7
JSON Schema Structured Output
(raw) {"hash": "bfbd5e7"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 846 in → 142 out (121 reasoning)
0e98025
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 1,010 in → 214 out (179 reasoning)
d3b4f6b
JSON Schema Structured Output
(raw) {"hash": "d3b4f6b"}
anthropic/claude-opus-4.6:high PASS 100% 814 in → 7 out (0 reasoning)
481df22
anthropic/claude-opus-4.6:high__json_schema PASS 100% 977 in → 10 out (0 reasoning)
246792e
JSON Schema Structured Output
(raw) {"hash":"246792e"}
anthropic/claude-opus-4.6:low PASS 100% 805 in → 8 out (0 reasoning)
8153dbe
anthropic/claude-opus-4.6:low__json_schema PASS 100% 967 in → 10 out (0 reasoning)
000ac59
JSON Schema Structured Output
(raw) {"hash":"000ac59"}
anthropic/claude-opus-4.6:max PASS 100% 791 in → 8 out (0 reasoning)
1255bd9
anthropic/claude-opus-4.6:max__json_schema PASS 100% 976 in → 81 out (51 reasoning)
95e5492
JSON Schema Structured Output
(raw) {"hash":"95e5492"}
anthropic/claude-opus-4.6:medium PASS 100% 797 in → 9 out (0 reasoning)
9f66ca1
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 956 in → 11 out (0 reasoning)
960bde8
JSON Schema Structured Output
(raw) {"hash":"960bde8"}
anthropic/claude-opus-4.6:none PASS 100% 818 in → 8 out (0 reasoning)
a583c5c
anthropic/claude-opus-4.6:none__json_schema PASS 100% 961 in → 11 out (0 reasoning)
caa7424
JSON Schema Structured Output
(raw) {"hash":"caa7424"}
anthropic/claude-opus-4.6:xhigh PASS 100% 819 in → 8 out (0 reasoning)
655a9dd
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 989 in → 11 out (0 reasoning)
31e945a
JSON Schema Structured Output
(raw) {"hash":"31e945a"}
anthropic/claude-opus-4.7:high PASS 100% 963 in → 8 out (0 reasoning)
f37982c
anthropic/claude-opus-4.7:high__json_schema PASS 100% 1,199 in → 15 out (0 reasoning)
815337d
JSON Schema Structured Output
(raw) {"hash":"815337d"}
anthropic/claude-opus-4.7:low PASS 100% 964 in → 8 out (0 reasoning)
777e109
anthropic/claude-opus-4.7:low__json_schema PASS 100% 1,183 in → 17 out (0 reasoning)
6840c7b
JSON Schema Structured Output
(raw) {"hash":"6840c7b"}
anthropic/claude-opus-4.7:max PASS 100% 992 in → 9 out (0 reasoning)
4207cea
anthropic/claude-opus-4.7:max__json_schema PASS 100% 1,191 in → 16 out (0 reasoning)
89600b3
JSON Schema Structured Output
(raw) {"hash":"89600b3"}
anthropic/claude-opus-4.7:medium PASS 100% 957 in → 8 out (0 reasoning)
68db346
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 1,191 in → 16 out (0 reasoning)
36a4029
JSON Schema Structured Output
(raw) {"hash":"36a4029"}
anthropic/claude-opus-4.7:none PASS 100% 980 in → 10 out (0 reasoning)
2fda49d
anthropic/claude-opus-4.7:none__json_schema PASS 100% 1,194 in → 17 out (0 reasoning)
e2375b3
JSON Schema Structured Output
(raw) {"hash":"e2375b3"}
anthropic/claude-opus-4.7:xhigh PASS 100% 977 in → 10 out (0 reasoning)
66fffa1
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 1,182 in → 18 out (0 reasoning)
8fd91b3
JSON Schema Structured Output
(raw) {"hash":"8fd91b3"}
anthropic/claude-opus-4.8:high PASS 100% 985 in → 7 out (0 reasoning)
882e8f9
anthropic/claude-opus-4.8:high__json_schema PASS 100% 1,184 in → 13 out (0 reasoning)
a503958
JSON Schema Structured Output
(raw) {"hash":"a503958"}
anthropic/claude-opus-4.8:low PASS 100% 965 in → 6 out (0 reasoning)
09fd285
anthropic/claude-opus-4.8:low__json_schema PASS 100% 1,162 in → 15 out (0 reasoning)
c35fb26
JSON Schema Structured Output
(raw) {"hash":"c35fb26"}
anthropic/claude-opus-4.8:max PASS 100% 967 in → 7 out (0 reasoning)
ee95afe
anthropic/claude-opus-4.8:max__json_schema PASS 100% 1,169 in → 15 out (0 reasoning)
e95e10e
JSON Schema Structured Output
(raw) {"hash":"e95e10e"}
anthropic/claude-opus-4.8:medium PASS 100% 985 in → 7 out (0 reasoning)
ce65de8
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 1,203 in → 14 out (0 reasoning)
f86a429
JSON Schema Structured Output
(raw) {"hash":"f86a429"}
anthropic/claude-opus-4.8:none PASS 100% 967 in → 7 out (0 reasoning)
845c7ba
anthropic/claude-opus-4.8:none__json_schema PASS 100% 1,186 in → 13 out (0 reasoning)
d808b49
JSON Schema Structured Output
(raw) {"hash":"d808b49"}
anthropic/claude-opus-4.8:xhigh PASS 100% 962 in → 9 out (0 reasoning)
d8e8bbe
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 1,186 in → 14 out (0 reasoning)
644f53b
JSON Schema Structured Output
(raw) {"hash":"644f53b"}
anthropic/claude-sonnet-4.6:high PASS 100% 804 in → 8 out (0 reasoning)
e879c1d
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 973 in → 11 out (0 reasoning)
d6ad708
JSON Schema Structured Output
(raw) {"hash":"d6ad708"}
anthropic/claude-sonnet-4.6:low PASS 100% 800 in → 9 out (0 reasoning)
1d2010f
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 973 in → 12 out (0 reasoning)
487b0e4
JSON Schema Structured Output
(raw) {"hash":"487b0e4"}
anthropic/claude-sonnet-4.6:max PASS 100% 812 in → 9 out (0 reasoning)
43f8cc3
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 982 in → 10 out (0 reasoning)
71dd490
JSON Schema Structured Output
(raw) {"hash":"71dd490"}
anthropic/claude-sonnet-4.6:medium PASS 100% 819 in → 7 out (0 reasoning)
134250b
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 977 in → 11 out (0 reasoning)
1510dd7
JSON Schema Structured Output
(raw) {"hash":"1510dd7"}
anthropic/claude-sonnet-4.6:none PASS 100% 802 in → 8 out (0 reasoning)
a6a8758
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 976 in → 12 out (0 reasoning)
5ef1e18
JSON Schema Structured Output
(raw) {"hash":"5ef1e18"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 790 in → 8 out (0 reasoning)
d15ccc4
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 954 in → 11 out (0 reasoning)
2480c82
JSON Schema Structured Output
(raw) {"hash":"2480c82"}
anthropic/claude-sonnet-5:high PASS 100% 964 in → 8 out (0 reasoning)
cdbce5e
anthropic/claude-sonnet-5:high__json_schema PASS 100% 1,153 in → 13 out (0 reasoning)
2444781
JSON Schema Structured Output
(raw) {"hash": "2444781"}
anthropic/claude-sonnet-5:low PASS 100% 979 in → 8 out (0 reasoning)
7bf2af5
anthropic/claude-sonnet-5:low__json_schema PASS 100% 1,182 in → 16 out (0 reasoning)
3ede4a4
JSON Schema Structured Output
(raw) {"hash":"3ede4a4"}
anthropic/claude-sonnet-5:medium PASS 100% 980 in → 7 out (0 reasoning)
997c3d4
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 1,180 in → 14 out (0 reasoning)
96b930f
JSON Schema Structured Output
(raw) {"hash": "96b930f"}
anthropic/claude-sonnet-5:none PASS 100% 991 in → 5 out (0 reasoning)
e640789
anthropic/claude-sonnet-5:none__json_schema PASS 100% 1,172 in → 14 out (0 reasoning)
e6d5219
JSON Schema Structured Output
(raw) {"hash":"e6d5219"}
anthropic/claude-sonnet-5:xhigh PASS 100% 1,007 in → 8 out (0 reasoning)
e67ef8c
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 1,194 in → 15 out (0 reasoning)
7720e0b
JSON Schema Structured Output
(raw) {"hash": "7720e0b"}
arcee-ai/trinity-large-thinking:high PASS 100% 693 in → 381 out (377 reasoning)
3b5f362
arcee-ai/trinity-large-thinking:low PASS 100% 702 in → 356 out (352 reasoning)
535da82
arcee-ai/trinity-large-thinking:medium PASS 100% 709 in → 323 out (316 reasoning)
3be0050
arcee-ai/trinity-large-thinking:xhigh PASS 100% 695 in → 319 out (313 reasoning)
f94279f
arcee-ai/trinity-mini:high PASS 100% 694 in → 254 out (248 reasoning)
d2959fb
arcee-ai/trinity-mini:high__json_schema PASS 100% 724 in → 240 out (234 reasoning)
a4346a5
JSON Schema Structured Output
(raw) { "hash":"a4346a5" }
arcee-ai/trinity-mini:low PASS 100% 703 in → 274 out (286 reasoning)
81daeba
arcee-ai/trinity-mini:low__json_schema PASS 100% 728 in → 209 out (198 reasoning)
eb6a5b0
JSON Schema Structured Output
(raw) { "hash": "eb6a5b0" }
arcee-ai/trinity-mini:medium PASS 100% 708 in → 257 out (242 reasoning)
e57cb9c
arcee-ai/trinity-mini:medium__json_schema PASS 100% 702 in → 335 out (310 reasoning)
e8861e2
JSON Schema Structured Output
(raw) { "hash": "e8861e2" }
arcee-ai/trinity-mini:xhigh PASS 100% 720 in → 241 out (232 reasoning)
fddbb5b
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 706 in → 239 out (237 reasoning)
3c9e099
JSON Schema Structured Output
(raw) { "hash": "3c9e099" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 794 in → 39 out (33 reasoning)
d802227
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 798 in → 31 out (19 reasoning)
b4385d8
JSON Schema Structured Output
(raw) {"hash": "b4385d8"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 784 in → 36 out (32 reasoning)
875a903
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 720 in → 65 out (53 reasoning)
2561146
JSON Schema Structured Output
(raw) { "hash": "2561146" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 800 in → 22 out (16 reasoning)
a20ef14
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 805 in → 62 out (44 reasoning)
39b6bcf
JSON Schema Structured Output
(raw) { "hash": "39b6bcf" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 721 in → 62 out (56 reasoning)
3d39e82
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 724 in → 59 out (42 reasoning)
9eb8037
JSON Schema Structured Output
(raw) { "hash": "9eb8037" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 735 in → 6 out (0 reasoning)
b1d43af
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 721 in → 14 out (0 reasoning)
27dd9c0
JSON Schema Structured Output
(raw) { "hash": "27dd9c0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 819 in → 38 out (31 reasoning)
c8643f1
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 708 in → 43 out (36 reasoning)
f5565bc
JSON Schema Structured Output
(raw) { "hash": "f5565bc" }
deepseek/deepseek-v4-flash:high PASS 100% 718 in → 97 out (90 reasoning)
3b7976a
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 792 in → 8 out (0 reasoning)
700a786
JSON Schema Structured Output
(raw) {"hash":"700a786"}
deepseek/deepseek-v4-flash:low PASS 100% 735 in → 64 out (59 reasoning)
5bca70a
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 720 in → 106 out (93 reasoning)
9239ab1
JSON Schema Structured Output
(raw) { "hash": "9239ab1" }
deepseek/deepseek-v4-flash:medium PASS 100% 704 in → 50 out (46 reasoning)
9219587
deepseek/deepseek-v4-flash:none PASS 100% 732 in → 4 out (0 reasoning)
d162774
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 812 in → 10 out (0 reasoning)
c9ddf61
JSON Schema Structured Output
(raw) {"hash":"c9ddf61"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 819 in → 118 out (112 reasoning)
2cae554
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 811 in → 144 out (135 reasoning)
be42437
JSON Schema Structured Output
(raw) { "hash": "be42437" }
deepseek/deepseek-v4-pro:high PASS 100% 722 in → 46 out (39 reasoning)
c0fdcad
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 711 in → 123 out (110 reasoning)
3658cca
JSON Schema Structured Output
(raw) { "hash": "3658cca" }
deepseek/deepseek-v4-pro:low PASS 100% 719 in → 51 out (45 reasoning)
7f32372
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 902 in → 260 out (247 reasoning)
3b68e2c
JSON Schema Structured Output
(raw) {"hash": "3b68e2c"}
deepseek/deepseek-v4-pro:medium PASS 100% 727 in → 86 out (79 reasoning)
3922e9d
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 702 in → 12 out (0 reasoning)
62e28cc
JSON Schema Structured Output
(raw) { "hash": "62e28cc" }
deepseek/deepseek-v4-pro:none PASS 100% 732 in → 4 out (0 reasoning)
39feecb
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 911 in → 11 out (0 reasoning)
40c3cba
JSON Schema Structured Output
(raw) {"hash": "40c3cba"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 802 in → 144 out (137 reasoning)
65fa7b2
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 960 in → 432 out (422 reasoning)
ee14ab9
JSON Schema Structured Output
(raw) {"hash":"ee14ab9"}
google/gemini-3-flash-preview:high PASS 100% 921 in → 194 out (189 reasoning)
fd44db3
google/gemini-3-flash-preview:high__json_schema PASS 100% 952 in → 295 out (285 reasoning)
54ec6c3
JSON Schema Structured Output
(raw) {"hash":"54ec6c3"}
google/gemini-3-flash-preview:low PASS 100% 914 in → 288 out (281 reasoning)
5750b6f
google/gemini-3-flash-preview:low__json_schema PASS 100% 961 in → 149 out (139 reasoning)
ea53538
JSON Schema Structured Output
(raw) {"hash":"ea53538"}
google/gemini-3-flash-preview:medium PASS 100% 929 in → 252 out (245 reasoning)
d616456
google/gemini-3-flash-preview:medium__json_schema PASS 100% 1,003 in → 339 out (323 reasoning)
85a1e6e
JSON Schema Structured Output
(raw) { "hash": "85a1e6e" }
google/gemini-3-flash-preview:xhigh PASS 100% 926 in → 246 out (240 reasoning)
a13aa8d
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 969 in → 108 out (96 reasoning)
98b902c
JSON Schema Structured Output
(raw) {"hash": "98b902c"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 927 in → 335 out (329 reasoning)
94af20e
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 926 in → 399 out (390 reasoning)
bdd6a95
JSON Schema Structured Output
(raw) {"hash":"bdd6a95"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 925 in → 127 out (122 reasoning)
b4b2eba
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 960 in → 134 out (118 reasoning)
80ab752
JSON Schema Structured Output
(raw) { "hash": "80ab752" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 913 in → 187 out (180 reasoning)
a40c710
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 930 in → 221 out (209 reasoning)
76f914f
JSON Schema Structured Output
(raw) {"hash": "76f914f"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 923 in → 296 out (290 reasoning)
43045fb
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 988 in → 227 out (215 reasoning)
67633c9
JSON Schema Structured Output
(raw) {"hash": "67633c9"}
google/gemini-3.1-pro-preview:high PASS 100% 935 in → 284 out (279 reasoning)
8ef29de
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 952 in → 100 out (92 reasoning)
ee04beb
JSON Schema Structured Output
(raw) {"hash":"ee04beb"}
google/gemini-3.1-pro-preview:low PASS 100% 938 in → 206 out (199 reasoning)
e572101
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 967 in → 90 out (79 reasoning)
b00dc25
JSON Schema Structured Output
(raw) {"hash": "b00dc25"}
google/gemini-3.1-pro-preview:medium PASS 100% 915 in → 229 out (223 reasoning)
751afc3
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 967 in → 88 out (77 reasoning)
167ee73
JSON Schema Structured Output
(raw) {"hash": "167ee73"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 921 in → 223 out (216 reasoning)
5078815
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 965 in → 317 out (308 reasoning)
abe2e9b
JSON Schema Structured Output
(raw) {"hash":"abe2e9b"}
google/gemini-3.5-flash-lite:high PASS 100% 931 in → 263 out (256 reasoning)
422e902
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 986 in → 406 out (395 reasoning)
24f46ad
JSON Schema Structured Output
(raw) {"hash": "24f46ad"}
google/gemini-3.5-flash-lite:low PASS 100% 934 in → 6 out (0 reasoning)
6ceb580
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 931 in → 17 out (0 reasoning)
4d76c63
JSON Schema Structured Output
(raw) { "hash": "4d76c63" }
google/gemini-3.5-flash-lite:max PASS 100% 930 in → 257 out (252 reasoning)
ec12ea6
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 973 in → 117 out (106 reasoning)
0946111
JSON Schema Structured Output
(raw) {"hash":"0946111"}
google/gemini-3.5-flash-lite:medium PASS 100% 931 in → 202 out (198 reasoning)
77aadbd
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 980 in → 323 out (313 reasoning)
9bd7b76
JSON Schema Structured Output
(raw) {"hash":"9bd7b76"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 924 in → 277 out (270 reasoning)
280d2b3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 972 in → 455 out (446 reasoning)
bf4cf2a
JSON Schema Structured Output
(raw) {"hash":"bf4cf2a"}
google/gemini-3.5-flash:high PASS 100% 928 in → 247 out (241 reasoning)
7bd26a0
google/gemini-3.5-flash:high__json_schema PASS 100% 970 in → 139 out (129 reasoning)
dd7a81e
JSON Schema Structured Output
(raw) {"hash":"dd7a81e"}
google/gemini-3.5-flash:low PASS 100% 935 in → 217 out (210 reasoning)
6c28f34
google/gemini-3.5-flash:low__json_schema PASS 100% 970 in → 78 out (62 reasoning)
b9dd784
JSON Schema Structured Output
(raw) { "hash": "b9dd784" }
google/gemini-3.5-flash:medium PASS 100% 943 in → 234 out (227 reasoning)
9e6d8f1
google/gemini-3.5-flash:medium__json_schema PASS 100% 996 in → 116 out (106 reasoning)
749e2bf
JSON Schema Structured Output
(raw) {"hash":"749e2bf"}
google/gemini-3.5-flash:xhigh PASS 100% 928 in → 334 out (327 reasoning)
774c90a
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 971 in → 148 out (137 reasoning)
33815c9
JSON Schema Structured Output
(raw) {"hash":"33815c9"}
google/gemini-3.6-flash:high PASS 100% 935 in → 283 out (276 reasoning)
e5b5355
google/gemini-3.6-flash:high__json_schema PASS 100% 928 in → 233 out (223 reasoning)
9f62db1
JSON Schema Structured Output
(raw) {"hash":"9f62db1"}
google/gemini-3.6-flash:low PASS 100% 945 in → 62 out (55 reasoning)
9a0e104
google/gemini-3.6-flash:low__json_schema PASS 100% 926 in → 11 out (0 reasoning)
a9c99b1
JSON Schema Structured Output
(raw) {"hash":"a9c99b1"}
google/gemini-3.6-flash:max PASS 100% 925 in → 268 out (261 reasoning)
8f38855
google/gemini-3.6-flash:max__json_schema PASS 100% 977 in → 214 out (203 reasoning)
67b108e
JSON Schema Structured Output
(raw) {"hash":"67b108e"}
google/gemini-3.6-flash:medium PASS 100% 932 in → 144 out (137 reasoning)
8576587
google/gemini-3.6-flash:medium__json_schema PASS 100% 977 in → 164 out (154 reasoning)
39ad843
JSON Schema Structured Output
(raw) {"hash":"39ad843"}
google/gemini-3.6-flash:xhigh PASS 100% 926 in → 80 out (73 reasoning)
d019461
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 933 in → 253 out (243 reasoning)
d7299fe
JSON Schema Structured Output
(raw) {"hash":"d7299fe"}
google/gemma-4-26b-a4b-it:high PASS 100% 954 in → 315 out (235 reasoning)
828f7b6
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 942 in → 326 out (222 reasoning)
119a954
JSON Schema Structured Output
(raw) { "hash": "119a954" }
google/gemma-4-26b-a4b-it:low PASS 100% 945 in → 322 out (213 reasoning)
05dd1c2
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 930 in → 398 out (270 reasoning)
2770c1c
JSON Schema Structured Output
(raw) { "hash": "2770c1c" }
google/gemma-4-26b-a4b-it:medium PASS 100% 910 in → 283 out (217 reasoning)
1feebe6
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 942 in → 429 out (269 reasoning)
957f40d
JSON Schema Structured Output
(raw) { "hash": "957f40d" }
google/gemma-4-26b-a4b-it:none PASS 100% 950 in → 6 out (0 reasoning)
eeec71d
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 958 in → 18 out (0 reasoning)
b71e01c
JSON Schema Structured Output
(raw) { "hash": "b71e01c" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 938 in → 230 out (172 reasoning)
e049804
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 935 in → 382 out (262 reasoning)
da66a9b
JSON Schema Structured Output
(raw) { "hash": "da66a9b" }
google/gemma-4-31b-it:high PASS 100% 948 in → 194 out (149 reasoning)
241bfd6
google/gemma-4-31b-it:high__json_schema PASS 100% 944 in → 193 out (139 reasoning)
29cbb7b
JSON Schema Structured Output
(raw) { "hash": "29cbb7b" }
google/gemma-4-31b-it:low PASS 100% 901 in → 203 out (1 reasoning)
fc772bf
google/gemma-4-31b-it:medium PASS 100% 944 in → 181 out (141 reasoning)
cbf56d3
google/gemma-4-31b-it:medium__json_schema PASS 100% 931 in → 229 out (165 reasoning)
a903998
JSON Schema Structured Output
(raw) { "hash": "a903998" }
google/gemma-4-31b-it:none PASS 100% 940 in → 8 out (0 reasoning)
f78e39f
google/gemma-4-31b-it:none__json_schema PASS 100% 921 in → 11 out (0 reasoning)
eadf5e8
JSON Schema Structured Output
(raw) {"hash": "eadf5e8"}
google/gemma-4-31b-it:xhigh PASS 100% 934 in → 186 out (1 reasoning)
a3ce9c8
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 924 in → 178 out (141 reasoning)
e2305fb
JSON Schema Structured Output
(raw) {"hash": "e2305fb"}
ibm-granite/granite-4.1-8b PASS 100% 756 in → 4 out
dad9176
ibm-granite/granite-4.1-8b__json_schema PASS 100% 765 in → 14 out
0c9e2fc
JSON Schema Structured Output
(raw) { "hash": "0c9e2fc" }
inclusionai/ling-2.6-flash PASS 100% 948 in → 7 out
13274fe
inclusionai/ling-2.6-flash__json_schema PASS 100% 962 in → 16 out
4923aeb
JSON Schema Structured Output
(raw) { "hash": "4923aeb" }
liquid/lfm-2-24b-a2b PASS 100% 749 in → 5 out
7dedaef
minimax/minimax-m2.5:high PASS 100% 757 in → 942 out (672 reasoning)
8dd779e
minimax/minimax-m2.5:high__json_schema PASS 100% 745 in → 115 out (109 reasoning)
65b267f
JSON Schema Structured Output
(raw) { "hash": "65b267f" }
minimax/minimax-m2.5:low PASS 100% 753 in → 104 out (99 reasoning)
c7daff8
minimax/minimax-m2.5:low__json_schema PASS 100% 754 in → 119 out (111 reasoning)
a4c4536
JSON Schema Structured Output
(raw) { "hash": "a4c4536" }
minimax/minimax-m2.5:medium PASS 100% 741 in → 160 out (154 reasoning)
68ba74b
minimax/minimax-m2.5:medium__json_schema PASS 100% 725 in → 152 out (139 reasoning)
67dfb05
JSON Schema Structured Output
(raw) { "hash": "67dfb05" }
minimax/minimax-m2.5:xhigh PASS 100% 751 in → 179 out (172 reasoning)
9b16e4e
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 748 in → 75 out (74 reasoning)
bb600dd
JSON Schema Structured Output
(raw) { "hash": "bb600dd" }
minimax/minimax-m2.7:high PASS 100% 722 in → 176 out (169 reasoning)
00ddd4c
minimax/minimax-m2.7:low PASS 100% 756 in → 132 out (127 reasoning)
782f2cb
minimax/minimax-m2.7:low__json_schema PASS 100% 858 in → 366 out (356 reasoning)
1873bb0
JSON Schema Structured Output
(raw) {"hash":"1873bb0"}
minimax/minimax-m2.7:medium PASS 100% 741 in → 223 out (217 reasoning)
9f7ede5
minimax/minimax-m2.7:medium__json_schema PASS 100% 891 in → 350 out (335 reasoning)
6acaf9a
JSON Schema Structured Output
(raw) { "hash": "6acaf9a" }
minimax/minimax-m2.7:xhigh PASS 100% 747 in → 168 out (160 reasoning)
4e1b1af
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 854 in → 288 out (278 reasoning)
e1aa425
JSON Schema Structured Output
(raw) {"hash":"e1aa425"}
minimax/minimax-m3:high PASS 100% 861 in → 61 out (53 reasoning)
cbc5a1d
minimax/minimax-m3:high__json_schema PASS 100% 871 in → 10 out (0 reasoning)
366f41f
JSON Schema Structured Output
(raw) {"hash": "366f41f"}
minimax/minimax-m3:low PASS 100% 875 in → 61 out (55 reasoning)
66dd06d
minimax/minimax-m3:low__json_schema PASS 100% 859 in → 13 out (0 reasoning)
1fdb835
JSON Schema Structured Output
(raw) { "hash": "1fdb835" }
minimax/minimax-m3:medium PASS 100% 868 in → 38 out (33 reasoning)
fdc5df9
minimax/minimax-m3:medium__json_schema PASS 100% 868 in → 12 out (0 reasoning)
1b23526
JSON Schema Structured Output
(raw) { "hash": "1b23526" }
minimax/minimax-m3:xhigh PASS 100% 865 in → 55 out (52 reasoning)
b927dfc
minimax/minimax-m3:xhigh__json_schema PASS 100% 870 in → 11 out (0 reasoning)
cc22514
JSON Schema Structured Output
(raw) { "hash": "cc22514" }
mistralai/devstral-2512 PASS 100% 919 in → 6 out
4889eca
mistralai/devstral-2512__json_schema PASS 100% 933 in → 13 out
b56c79b
JSON Schema Structured Output
(raw) {"hash": "b56c79b"}
mistralai/mistral-medium-3-5:high PASS 100% 932 in → 322 out (264 reasoning)
26d4088
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 897 in → 667 out (656 reasoning)
89f394a
JSON Schema Structured Output
(raw) {"hash": "89f394a"}
mistralai/mistral-medium-3-5:low PASS 100% 915 in → 300 out (239 reasoning)
b3c7304
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 910 in → 263 out (205 reasoning)
c2bd4f7
JSON Schema Structured Output
(raw) {"hash": "c2bd4f7"}
mistralai/mistral-medium-3-5:medium PASS 100% 920 in → 194 out (165 reasoning)
63190e8
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 907 in → 490 out (487 reasoning)
6317b54
JSON Schema Structured Output
(raw) {"hash": "6317b54"}
mistralai/mistral-medium-3-5:none PASS 100% 919 in → 8 out (0 reasoning)
5f11c1a
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 912 in → 13 out (0 reasoning)
a713866
JSON Schema Structured Output
(raw) {"hash": "a713866"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 919 in → 209 out (165 reasoning)
0e233ac
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 909 in → 328 out (310 reasoning)
7ed9993
JSON Schema Structured Output
(raw) {"hash": "7ed9993"}
moonshotai/kimi-k2.6:high PASS 100% 699 in → 162 out (155 reasoning)
570f7a2
moonshotai/kimi-k2.6:high__json_schema PASS 100% 702 in → 119 out (110 reasoning)
f2828c2
JSON Schema Structured Output
(raw) {"hash":"f2828c2"}
moonshotai/kimi-k2.6:low PASS 100% 710 in → 119 out (109 reasoning)
457bef7
moonshotai/kimi-k2.6:medium PASS 100% 704 in → 126 out (138 reasoning)
c170e38
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 698 in → 129 out (118 reasoning)
75c4d77
JSON Schema Structured Output
(raw) {"hash": "75c4d77"}
moonshotai/kimi-k2.6:xhigh PASS 100% 688 in → 121 out (127 reasoning)
f021b23
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 700 in → 145 out (130 reasoning)
5d49eb5
JSON Schema Structured Output
(raw) { "hash": "5d49eb5" }
moonshotai/kimi-k2.7-code:high PASS 100% 729 in → 104 out (99 reasoning)
c6fb45e
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 695 in → 58 out (45 reasoning)
e3db733
JSON Schema Structured Output
(raw) { "hash": "e3db733" }
moonshotai/kimi-k2.7-code:low PASS 100% 704 in → 28 out (17 reasoning)
91ca5a9
moonshotai/kimi-k2.7-code:medium PASS 100% 721 in → 140 out (132 reasoning)
bf6e2e7
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 702 in → 60 out (41 reasoning)
4e1ec20
JSON Schema Structured Output
(raw) { "hash": "4e1ec20" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 728 in → 157 out (151 reasoning)
d32decb
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 710 in → 56 out (37 reasoning)
30a8c44
JSON Schema Structured Output
(raw) { "hash": "30a8c44" }
moonshotai/kimi-k3:high PASS 100% 760 in → 70 out (52 reasoning)
d206173
moonshotai/kimi-k3:high__json_schema PASS 100% 786 in → 64 out (45 reasoning)
5e657bf
moonshotai/kimi-k3:low PASS 100% 764 in → 108 out (90 reasoning)
7028962
moonshotai/kimi-k3:low__json_schema PASS 100% 802 in → 78 out (58 reasoning)
ce2ee1d
moonshotai/kimi-k3:max PASS 100% 772 in → 67 out (49 reasoning)
47aa661
moonshotai/kimi-k3:max__json_schema PASS 100% 769 in → 62 out (43 reasoning)
ebcfde2
moonshotai/kimi-k3:medium PASS 100% 783 in → 68 out (48 reasoning)
df7a91d
moonshotai/kimi-k3:medium__json_schema PASS 100% 771 in → 70 out (50 reasoning)
7bdc7b4
moonshotai/kimi-k3:xhigh PASS 100% 793 in → 65 out (47 reasoning)
95038bb
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 792 in → 71 out (51 reasoning)
35a9ea8
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 929 in → 146 out (98 reasoning)
a56c86b
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 923 in → 168 out (129 reasoning)
a9c3228
JSON Schema Structured Output
(raw) { "hash": "a9c3228" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 942 in → 153 out (127 reasoning)
54b4883
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 922 in → 98 out (75 reasoning)
cbc590f
JSON Schema Structured Output
(raw) { "hash": "cbc590f" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 926 in → 156 out (140 reasoning)
dc1ab91
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 918 in → 144 out (117 reasoning)
ae4e51c
JSON Schema Structured Output
(raw) { "hash": "ae4e51c" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 936 in → 8 out (0 reasoning)
4539f6c
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 936 in → 14 out (0 reasoning)
9ff66a6
JSON Schema Structured Output
(raw) { "hash": "9ff66a6" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 927 in → 86 out (73 reasoning)
79fa2f5
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 923 in → 169 out (137 reasoning)
55f7146
JSON Schema Structured Output
(raw) {"hash":"55f7146" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 934 in → 67 out (51 reasoning)
691865d
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 910 in → 111 out (87 reasoning)
b02fa5d
JSON Schema Structured Output
(raw) { "hash": "b02fa5d" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 909 in → 57 out (47 reasoning)
c10ade4
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 933 in → 66 out (43 reasoning)
f582e86
JSON Schema Structured Output
(raw) { "hash": "f582e86" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 935 in → 126 out (107 reasoning)
421eda8
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 930 in → 113 out (84 reasoning)
03e7dda
JSON Schema Structured Output
(raw) { "hash": "03e7dda" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 930 in → 8 out (0 reasoning)
8a62962
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 926 in → 16 out (0 reasoning)
8b74c45
JSON Schema Structured Output
(raw) { "hash": "8b74c45" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 919 in → 68 out (56 reasoning)
44c6c4c
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 935 in → 634 out (548 reasoning)
8c0555f
JSON Schema Structured Output
(raw) { "hash": "8c0555f" }
openai/gpt-5.4-mini:high PASS 100% 726 in → 48 out (38 reasoning)
825ba1a
openai/gpt-5.4-mini:high__json_schema PASS 100% 751 in → 50 out (32 reasoning)
f87b353
JSON Schema Structured Output
(raw) {"hash":"f87b353"}
openai/gpt-5.4-mini:low PASS 100% 712 in → 31 out (22 reasoning)
893a308
openai/gpt-5.4-mini:low__json_schema PASS 100% 736 in → 36 out (19 reasoning)
58327cd
JSON Schema Structured Output
(raw) {"hash":"58327cd"}
openai/gpt-5.4-mini:medium PASS 100% 696 in → 35 out (26 reasoning)
aecc424
openai/gpt-5.4-mini:medium__json_schema PASS 100% 727 in → 49 out (31 reasoning)
71b903b
JSON Schema Structured Output
(raw) {"hash":"71b903b"}
openai/gpt-5.4-mini:none PASS 100% 710 in → 7 out (0 reasoning)
ec48658
openai/gpt-5.4-mini:none__json_schema PASS 100% 727 in → 15 out (0 reasoning)
f135640
JSON Schema Structured Output
(raw) {"hash":"f135640"}
openai/gpt-5.4-mini:xhigh PASS 100% 727 in → 60 out (50 reasoning)
a85c328
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 736 in → 1,570 out (1,552 reasoning)
2fb0642
JSON Schema Structured Output
(raw) {"hash":"2fb0642"}
openai/gpt-5.4-nano:high PASS 100% 720 in → 8 out (0 reasoning)
6fb8083
openai/gpt-5.4-nano:high__json_schema PASS 100% 743 in → 16 out (0 reasoning)
30f89bc
JSON Schema Structured Output
(raw) {"hash":"30f89bc"}
openai/gpt-5.4-nano:low PASS 100% 697 in → 9 out (0 reasoning)
7fef86d
openai/gpt-5.4-nano:low__json_schema PASS 100% 736 in → 15 out (0 reasoning)
15894ac
JSON Schema Structured Output
(raw) {"hash":"15894ac"}
openai/gpt-5.4-nano:medium PASS 100% 696 in → 8 out (0 reasoning)
87228e7
openai/gpt-5.4-nano:medium__json_schema PASS 100% 735 in → 16 out (0 reasoning)
41b9833
JSON Schema Structured Output
(raw) {"hash":"41b9833"}
openai/gpt-5.4-nano:none PASS 100% 716 in → 9 out (0 reasoning)
7d0388e
openai/gpt-5.4-nano:none__json_schema PASS 100% 730 in → 17 out (0 reasoning)
b7139b0
JSON Schema Structured Output
(raw) {"hash":"b7139b0"}
openai/gpt-5.4-nano:xhigh PASS 100% 720 in → 43 out (34 reasoning)
325bc59
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 751 in → 78 out (60 reasoning)
85f2649
JSON Schema Structured Output
(raw) {"hash":"85f2649"}
openai/gpt-5.4:high PASS 100% 714 in → 39 out (28 reasoning)
de45d8f
openai/gpt-5.4:high__json_schema PASS 100% 754 in → 48 out (30 reasoning)
2cb459b
JSON Schema Structured Output
(raw) {"hash":"2cb459b"}
openai/gpt-5.4:low PASS 100% 715 in → 9 out (0 reasoning)
34bb4a9
openai/gpt-5.4:low__json_schema PASS 100% 760 in → 17 out (0 reasoning)
f6688d3
JSON Schema Structured Output
(raw) {"hash":"f6688d3"}
openai/gpt-5.4:medium PASS 100% 706 in → 8 out (0 reasoning)
b9cbcda
openai/gpt-5.4:medium__json_schema PASS 100% 748 in → 31 out (11 reasoning)
75b0e9e
JSON Schema Structured Output
(raw) {"hash":"75b0e9e"}
openai/gpt-5.4:none PASS 100% 710 in → 9 out (0 reasoning)
e26e82e
openai/gpt-5.4:none__json_schema PASS 100% 749 in → 17 out (0 reasoning)
f89cde6
JSON Schema Structured Output
(raw) {"hash":"f89cde6"}
openai/gpt-5.4:xhigh PASS 100% 708 in → 49 out (40 reasoning)
e279816
openai/gpt-5.4:xhigh__json_schema PASS 100% 744 in → 64 out (45 reasoning)
79b9b61
JSON Schema Structured Output
(raw) {"hash":"79b9b61"}
openai/gpt-5.5:high PASS 100% 697 in → 24 out (14 reasoning)
03cbe76
openai/gpt-5.5:high__json_schema PASS 100% 742 in → 16 out (0 reasoning)
c60e408
JSON Schema Structured Output
(raw) {"hash":"c60e408"}
openai/gpt-5.5:low PASS 100% 702 in → 8 out (0 reasoning)
e62ca17
openai/gpt-5.5:low__json_schema PASS 100% 766 in → 17 out (0 reasoning)
e8568a4
JSON Schema Structured Output
(raw) {"hash":"e8568a4"}
openai/gpt-5.5:medium PASS 100% 708 in → 9 out (0 reasoning)
4b4c462
openai/gpt-5.5:medium__json_schema PASS 100% 755 in → 18 out (0 reasoning)
4de7a1a
JSON Schema Structured Output
(raw) {"hash":"4de7a1a"}
openai/gpt-5.5:none PASS 100% 710 in → 8 out (0 reasoning)
9c87477
openai/gpt-5.5:none__json_schema PASS 100% 753 in → 16 out (0 reasoning)
dc6001c
JSON Schema Structured Output
(raw) {"hash":"dc6001c"}
openai/gpt-5.5:xhigh PASS 100% 693 in → 74 out (65 reasoning)
e490968
openai/gpt-5.5:xhigh__json_schema PASS 100% 728 in → 138 out (120 reasoning)
37b2331
JSON Schema Structured Output
(raw) {"hash":"37b2331"}
openai/gpt-5.6-luna:high PASS 100% 721 in → 9 out (0 reasoning)
4c9453e
openai/gpt-5.6-luna:high__json_schema PASS 100% 720 in → 16 out (0 reasoning)
ed5f320
JSON Schema Structured Output
(raw) {"hash":"ed5f320"}
openai/gpt-5.6-luna:low PASS 100% 700 in → 9 out (0 reasoning)
0a4baa9
openai/gpt-5.6-luna:low__json_schema PASS 100% 736 in → 16 out (0 reasoning)
f925bf7
JSON Schema Structured Output
(raw) {"hash":"f925bf7"}
openai/gpt-5.6-luna:max PASS 100% 696 in → 7 out (0 reasoning)
965157b
openai/gpt-5.6-luna:max__json_schema PASS 100% 736 in → 16 out (0 reasoning)
b42c714
JSON Schema Structured Output
(raw) {"hash":"b42c714"}
openai/gpt-5.6-luna:medium PASS 100% 712 in → 9 out (0 reasoning)
6fe08d3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 738 in → 16 out (0 reasoning)
b851c67
JSON Schema Structured Output
(raw) {"hash":"b851c67"}
openai/gpt-5.6-luna:none PASS 100% 713 in → 10 out (0 reasoning)
6cda0b4
openai/gpt-5.6-luna:none__json_schema PASS 100% 730 in → 16 out (0 reasoning)
fe7f778
JSON Schema Structured Output
(raw) {"hash":"fe7f778"}
openai/gpt-5.6-luna:xhigh PASS 100% 709 in → 8 out (0 reasoning)
75cd1ba
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 731 in → 17 out (0 reasoning)
f30c56d
JSON Schema Structured Output
(raw) {"hash":"f30c56d"}
openai/gpt-5.6-sol:high PASS 100% 712 in → 8 out (0 reasoning)
42cfc31
openai/gpt-5.6-sol:high__json_schema PASS 100% 733 in → 15 out (0 reasoning)
751ab33
JSON Schema Structured Output
(raw) {"hash":"751ab33"}
openai/gpt-5.6-sol:low PASS 100% 724 in → 9 out (0 reasoning)
6b06e42
openai/gpt-5.6-sol:low__json_schema PASS 100% 739 in → 17 out (0 reasoning)
f4d862f
JSON Schema Structured Output
(raw) {"hash":"f4d862f"}
openai/gpt-5.6-sol:max PASS 100% 707 in → 8 out (0 reasoning)
841c6fa
openai/gpt-5.6-sol:max__json_schema PASS 100% 742 in → 17 out (0 reasoning)
3db3ad4
JSON Schema Structured Output
(raw) {"hash":"3db3ad4"}
openai/gpt-5.6-sol:medium PASS 100% 717 in → 8 out (0 reasoning)
7390cd8
openai/gpt-5.6-sol:medium__json_schema PASS 100% 733 in → 17 out (0 reasoning)
c6cace1
JSON Schema Structured Output
(raw) {"hash":"c6cace1"}
openai/gpt-5.6-sol:none PASS 100% 709 in → 8 out (0 reasoning)
9c36915
openai/gpt-5.6-sol:none__json_schema PASS 100% 752 in → 18 out (0 reasoning)
d5a0c61
JSON Schema Structured Output
(raw) {"hash":"d5a0c61"}
openai/gpt-5.6-sol:xhigh PASS 100% 710 in → 9 out (0 reasoning)
ed9d2da
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 745 in → 16 out (0 reasoning)
3fcc0fc
JSON Schema Structured Output
(raw) {"hash":"3fcc0fc"}
openai/gpt-5.6-terra:high PASS 100% 706 in → 9 out (0 reasoning)
0184f9e
openai/gpt-5.6-terra:high__json_schema PASS 100% 751 in → 15 out (0 reasoning)
e280639
JSON Schema Structured Output
(raw) {"hash":"e280639"}
openai/gpt-5.6-terra:low PASS 100% 705 in → 8 out (0 reasoning)
fe3ab50
openai/gpt-5.6-terra:low__json_schema PASS 100% 738 in → 16 out (0 reasoning)
5c08269
JSON Schema Structured Output
(raw) {"hash":"5c08269"}
openai/gpt-5.6-terra:max PASS 100% 725 in → 9 out (0 reasoning)
571e7d2
openai/gpt-5.6-terra:max__json_schema PASS 100% 755 in → 17 out (0 reasoning)
15f3b30
JSON Schema Structured Output
(raw) {"hash":"15f3b30"}
openai/gpt-5.6-terra:medium PASS 100% 718 in → 8 out (0 reasoning)
7aeb714
openai/gpt-5.6-terra:medium__json_schema PASS 100% 710 in → 17 out (0 reasoning)
4cf1a78
JSON Schema Structured Output
(raw) {"hash":"4cf1a78"}
openai/gpt-5.6-terra:none PASS 100% 712 in → 7 out (0 reasoning)
0645535
openai/gpt-5.6-terra:none__json_schema PASS 100% 733 in → 17 out (0 reasoning)
0d261a1
JSON Schema Structured Output
(raw) {"hash":"0d261a1"}
openai/gpt-5.6-terra:xhigh PASS 100% 694 in → 9 out (0 reasoning)
0bdb13e
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 738 in → 18 out (0 reasoning)
a7e0f32
JSON Schema Structured Output
(raw) {"hash":"a7e0f32"}
openai/gpt-oss-120b:high PASS 100% 763 in → 62 out (48 reasoning)
acf0b75
openai/gpt-oss-120b:high__json_schema PASS 100% 758 in → 252 out (235 reasoning)
956df42
JSON Schema Structured Output
(raw) {"hash":"956df42"}
openai/gpt-oss-120b:low PASS 100% 778 in → 32 out (20 reasoning)
6ba9dad
openai/gpt-oss-120b:low__json_schema PASS 100% 763 in → 137 out (118 reasoning)
db88b4a
JSON Schema Structured Output
(raw) {"hash":"db88b4a"}
openai/gpt-oss-120b:medium PASS 100% 775 in → 46 out (43 reasoning)
be47584
openai/gpt-oss-120b:medium__json_schema PASS 100% 785 in → 132 out (105 reasoning)
8d8fd72
JSON Schema Structured Output
(raw) { "hash": "8d8fd72" }
openai/gpt-oss-120b:xhigh PASS 100% 768 in → 193 out (180 reasoning)
60c43aa
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 771 in → 254 out (226 reasoning)
4135a18
JSON Schema Structured Output
(raw) { "hash": "4135a18" }
openai/gpt-oss-20b:high PASS 100% 769 in → 458 out (404 reasoning)
bf1a9b4
openai/gpt-oss-20b:high__json_schema PASS 100% 764 in → 235 out (217 reasoning)
90f9d75
JSON Schema Structured Output
(raw) { "hash": "90f9d75" }
openai/gpt-oss-20b:low PASS 100% 771 in → 26 out (8 reasoning)
4a301e5
openai/gpt-oss-20b:medium PASS 100% 766 in → 47 out (30 reasoning)
f7947cf
openai/gpt-oss-20b:xhigh PASS 100% 748 in → 261 out (256 reasoning)
5298943
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 763 in → 218 out (196 reasoning)
46d01e0
JSON Schema Structured Output
(raw) {"hash":"46d01e0"}
poolside/laguna-m.1:high PASS 100% 917 in → 227 out (218 reasoning)
c16b1fd
poolside/laguna-m.1:high__json_schema PASS 100% 910 in → 303 out (284 reasoning)
7717572
JSON Schema Structured Output
(raw) { "hash" : "7717572" }
poolside/laguna-m.1:low PASS 100% 905 in → 286 out (278 reasoning)
fe2f4bb
poolside/laguna-m.1:low__json_schema PASS 100% 902 in → 269 out (255 reasoning)
e649c3c
JSON Schema Structured Output
(raw) {"hash": "e649c3c"}
poolside/laguna-m.1:medium PASS 100% 920 in → 252 out (242 reasoning)
9520c13
poolside/laguna-m.1:medium__json_schema PASS 100% 907 in → 275 out (261 reasoning)
6ebeefc
JSON Schema Structured Output
(raw) { "hash": "6ebeefc" }
poolside/laguna-m.1:none PASS 100% 921 in → 8 out (0 reasoning)
0ab0f87
poolside/laguna-m.1:none__json_schema PASS 100% 898 in → 18 out (0 reasoning)
48983b9
JSON Schema Structured Output
(raw) { "hash": "48983b9" }
poolside/laguna-m.1:xhigh PASS 100% 912 in → 197 out (187 reasoning)
0a7206c
poolside/laguna-m.1:xhigh__json_schema PASS 100% 911 in → 346 out (328 reasoning)
03e7818
JSON Schema Structured Output
(raw) { "hash": "03e7818" }
poolside/laguna-xs-2.1:high PASS 100% 917 in → 269 out (262 reasoning)
f4eb71c
poolside/laguna-xs-2.1:high__json_schema PASS 100% 925 in → 477 out (458 reasoning)
2289606
JSON Schema Structured Output
(raw) { "hash": "2289606" }
poolside/laguna-xs-2.1:low PASS 100% 904 in → 281 out (273 reasoning)
b88e45d
poolside/laguna-xs-2.1:low__json_schema PASS 100% 907 in → 257 out (240 reasoning)
bd90584
JSON Schema Structured Output
(raw) { "hash": "bd90584" }
poolside/laguna-xs-2.1:medium PASS 100% 923 in → 370 out (362 reasoning)
616c30b
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 920 in → 309 out (291 reasoning)
f6de262
JSON Schema Structured Output
(raw) { "hash": "f6de262" }
poolside/laguna-xs-2.1:none PASS 100% 922 in → 7 out (0 reasoning)
4769b74
poolside/laguna-xs-2.1:none__json_schema PASS 100% 920 in → 12 out (0 reasoning)
87387cf
JSON Schema Structured Output
(raw) {"hash": "87387cf"}
poolside/laguna-xs-2.1:xhigh PASS 100% 906 in → 338 out (331 reasoning)
cd650a3
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 894 in → 311 out (292 reasoning)
918657b
JSON Schema Structured Output
(raw) { "hash": "918657b" }
poolside/laguna-xs.2:high PASS 100% 954 in → 146 out (136 reasoning)
585197d
poolside/laguna-xs.2:high__json_schema PASS 100% 939 in → 139 out (122 reasoning)
f1aff74
JSON Schema Structured Output
(raw) { "hash": "f1aff74" }
poolside/laguna-xs.2:low PASS 100% 955 in → 133 out (124 reasoning)
cb41234
poolside/laguna-xs.2:low__json_schema PASS 100% 941 in → 155 out (138 reasoning)
acec39b
JSON Schema Structured Output
(raw) { "hash": "acec39b" }
poolside/laguna-xs.2:medium PASS 100% 966 in → 149 out (139 reasoning)
2d26d31
poolside/laguna-xs.2:medium__json_schema PASS 100% 937 in → 151 out (135 reasoning)
ccadab4
JSON Schema Structured Output
(raw) { "hash": "ccadab4" }
poolside/laguna-xs.2:none PASS 100% 949 in → 9 out (0 reasoning)
2c0b3f8
poolside/laguna-xs.2:none__json_schema PASS 100% 957 in → 13 out (0 reasoning)
8e93769
JSON Schema Structured Output
(raw) {"hash": "8e93769"}
poolside/laguna-xs.2:xhigh PASS 100% 959 in → 135 out (125 reasoning)
84650b4
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 935 in → 147 out (135 reasoning)
ccd98f5
JSON Schema Structured Output
(raw) {"hash": "ccd98f5"}
qwen/qwen3.6-27b:high PASS 100% 921 in → 182 out (148 reasoning)
e935119
qwen/qwen3.6-27b:high__json_schema PASS 100% 898 in → 199 out (162 reasoning)
6ca8880
JSON Schema Structured Output
(raw) { "hash": "6ca8880" }
qwen/qwen3.6-27b:low PASS 100% 912 in → 181 out (155 reasoning)
0642966
qwen/qwen3.6-27b:low__json_schema PASS 100% 916 in → 18 out (0 reasoning)
a34c072
JSON Schema Structured Output
(raw) { "hash": "a34c072" }
qwen/qwen3.6-27b:medium PASS 100% 902 in → 189 out (1 reasoning)
074a895
qwen/qwen3.6-27b:none PASS 100% 903 in → 6 out (0 reasoning)
39feecb
qwen/qwen3.6-27b:none__json_schema PASS 100% 914 in → 17 out (0 reasoning)
20aa76b
JSON Schema Structured Output
(raw) { "hash": "20aa76b" }
qwen/qwen3.6-27b:xhigh PASS 100% 909 in → 129 out (118 reasoning)
4dd3e7a
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 902 in → 407 out (325 reasoning)
8566d51
JSON Schema Structured Output
(raw) { "hash": "8566d51" }
qwen/qwen3.6-35b-a3b:high PASS 100% 916 in → 221 out (209 reasoning)
1306208
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 928 in → 182 out (141 reasoning)
d7083a7
JSON Schema Structured Output
(raw) { "hash": "d7083a7" }
qwen/qwen3.6-35b-a3b:low PASS 100% 935 in → 169 out (158 reasoning)
685ca66
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 913 in → 260 out (212 reasoning)
78ba101
JSON Schema Structured Output
(raw) { "hash": "78ba101" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 917 in → 206 out (158 reasoning)
5176797
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 910 in → 183 out (155 reasoning)
cac135c
JSON Schema Structured Output
(raw) { "hash": "cac135c" }
qwen/qwen3.6-35b-a3b:none PASS 100% 914 in → 8 out (0 reasoning)
4427b45
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 923 in → 18 out (0 reasoning)
9990046
JSON Schema Structured Output
(raw) { "hash": "9990046" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 906 in → 212 out (201 reasoning)
87ae836
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 935 in → 277 out (207 reasoning)
c019672
JSON Schema Structured Output
(raw) { "hash": "c019672" }
qwen/qwen3.6-flash:high PASS 100% 910 in → 187 out (175 reasoning)
2b81756
qwen/qwen3.6-flash:low PASS 100% 924 in → 276 out (265 reasoning)
49950dc
qwen/qwen3.6-flash:medium PASS 100% 915 in → 352 out (340 reasoning)
0d79314
qwen/qwen3.6-flash:none PASS 100% 914 in → 7 out (0 reasoning)
7c56c6c
qwen/qwen3.6-flash:xhigh PASS 100% 923 in → 253 out (241 reasoning)
7233e86
qwen/qwen3.7-flash:high PASS 100% 926 in → 127 out (116 reasoning)
f50a8ba
qwen/qwen3.7-flash:low PASS 100% 902 in → 178 out (167 reasoning)
ad545f5
qwen/qwen3.7-flash:max PASS 100% 894 in → 157 out (147 reasoning)
ea3bce3
qwen/qwen3.7-flash:medium PASS 100% 917 in → 337 out (325 reasoning)
50b93e3
qwen/qwen3.7-flash:none PASS 100% 929 in → 5 out (0 reasoning)
7dba374
qwen/qwen3.7-flash:xhigh PASS 100% 912 in → 203 out (192 reasoning)
47fb04c
qwen/qwen3.7-max:high PASS 100% 939 in → 160 out (149 reasoning)
6f394ad
qwen/qwen3.7-max:low PASS 100% 909 in → 246 out (235 reasoning)
51f7fa4
qwen/qwen3.7-max:medium PASS 100% 919 in → 160 out (148 reasoning)
838d162
qwen/qwen3.7-max:none PASS 100% 919 in → 6 out (0 reasoning)
d4cd616
qwen/qwen3.7-max:xhigh PASS 100% 918 in → 137 out (126 reasoning)
d552db3
qwen/qwen3.7-plus:high PASS 100% 919 in → 139 out (127 reasoning)
c314b2d
qwen/qwen3.7-plus:low PASS 100% 921 in → 91 out (79 reasoning)
57f558f
qwen/qwen3.7-plus:medium PASS 100% 918 in → 181 out (170 reasoning)
7f5f8ae
qwen/qwen3.7-plus:none PASS 100% 911 in → 7 out (0 reasoning)
940e919
qwen/qwen3.7-plus:xhigh PASS 100% 895 in → 181 out (171 reasoning)
8fd4ec7
tencent/hy3:high PASS 100% 742 in → 73 out (66 reasoning)
cc9f7cc
tencent/hy3:high__json_schema PASS 100% 730 in → 79 out (66 reasoning)
f2df942
JSON Schema Structured Output
(raw) { "hash": "f2df942" }
tencent/hy3:low PASS 100% 721 in → 131 out (126 reasoning)
d310223
tencent/hy3:low__json_schema PASS 100% 722 in → 174 out (140 reasoning)
ce564b2
JSON Schema Structured Output
(raw) { "hash": "ce564b2" }
tencent/hy3:medium PASS 100% 715 in → 141 out (136 reasoning)
9068344
tencent/hy3:medium__json_schema PASS 100% 726 in → 144 out (125 reasoning)
ae8ca91
JSON Schema Structured Output
(raw) { "hash": "ae8ca91" }
tencent/hy3:none PASS 100% 729 in → 5 out (0 reasoning)
aae1927
tencent/hy3:none__json_schema PASS 100% 743 in → 12 out (0 reasoning)
c75f209
JSON Schema Structured Output
(raw) { "hash": "c75f209" }
tencent/hy3:xhigh PASS 100% 748 in → 186 out (161 reasoning)
29d2a5f
tencent/hy3:xhigh__json_schema PASS 100% 722 in → 130 out (105 reasoning)
3f45d3b
JSON Schema Structured Output
(raw) { "hash": "3f45d3b" }
thinkingmachines/inkling-small:high PASS 100% 731 in → 73 out (65 reasoning)
35cbcca
thinkingmachines/inkling-small:high__json_schema PASS 100% 710 in → 26 out (12 reasoning)
5db542a
JSON Schema Structured Output
(raw) {"hash":"5db542a"}
thinkingmachines/inkling-small:low PASS 100% 722 in → 61 out (52 reasoning)
ec74694
thinkingmachines/inkling-small:low__json_schema PASS 100% 756 in → 24 out (9 reasoning)
b84cd3a
JSON Schema Structured Output
(raw) {"hash": "b84cd3a"}
thinkingmachines/inkling-small:max PASS 100% 709 in → 76 out (65 reasoning)
d1655f6
thinkingmachines/inkling-small:max__json_schema PASS 100% 719 in → 79 out (61 reasoning)
6b8169a
JSON Schema Structured Output
(raw) { "hash": "6b8169a" }
thinkingmachines/inkling-small:medium PASS 100% 717 in → 21 out (9 reasoning)
c523e89
thinkingmachines/inkling-small:medium__json_schema PASS 100% 721 in → 78 out (62 reasoning)
3a149ed
JSON Schema Structured Output
(raw) { "hash": "3a149ed" }
thinkingmachines/inkling-small:none PASS 100% 704 in → 9 out (0 reasoning)
f02f7f4<|end_message|>
thinkingmachines/inkling-small:xhigh PASS 100% 717 in → 67 out (57 reasoning)
bf7ecc7
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 734 in → 71 out (55 reasoning)
2327378
JSON Schema Structured Output
(raw) { "hash": "2327378" }
thinkingmachines/inkling:high PASS 100% 707 in → 89 out (78 reasoning)
de075f5
thinkingmachines/inkling:high__json_schema PASS 100% 709 in → 41 out (25 reasoning)
42afd88
JSON Schema Structured Output
(raw) { "hash": "42afd88" }
thinkingmachines/inkling:low PASS 100% 716 in → 30 out (18 reasoning)
7fbb75b
thinkingmachines/inkling:max PASS 100% 705 in → 102 out (92 reasoning)
aef43ca
thinkingmachines/inkling:max__json_schema PASS 100% 710 in → 101 out (86 reasoning)
770240f
JSON Schema Structured Output
(raw) { "hash": "770240f" }
thinkingmachines/inkling:medium PASS 100% 724 in → 85 out (66 reasoning)
5e21d43
thinkingmachines/inkling:medium__json_schema PASS 100% 708 in → 49 out (33 reasoning)
fb727bc
JSON Schema Structured Output
(raw) { "hash": "fb727bc" }
thinkingmachines/inkling:none PASS 100% 692 in → 7 out (0 reasoning)
43fe738
thinkingmachines/inkling:none__json_schema PASS 100% 708 in → 13 out (0 reasoning)
d14b580
JSON Schema Structured Output
(raw) { "hash": "d14b580" }
thinkingmachines/inkling:xhigh PASS 100% 704 in → 104 out (94 reasoning)
f211769
x-ai/grok-4.3:high PASS 100% 881 in → 136 out (131 reasoning)
6b9d334
x-ai/grok-4.3:high__json_schema PASS 100% 931 in → 272 out (263 reasoning)
8e1948d
JSON Schema Structured Output
(raw) {"hash":"8e1948d"}
x-ai/grok-4.3:low PASS 100% 889 in → 185 out (181 reasoning)
9f87352
x-ai/grok-4.3:low__json_schema PASS 100% 941 in → 239 out (229 reasoning)
c7e60d7
JSON Schema Structured Output
(raw) {"hash":"c7e60d7"}
x-ai/grok-4.3:max PASS 100% 886 in → 193 out (189 reasoning)
540a84d
x-ai/grok-4.3:max__json_schema PASS 100% 930 in → 300 out (292 reasoning)
17dafa0
JSON Schema Structured Output
(raw) {"hash":"17dafa0"}
x-ai/grok-4.3:medium PASS 100% 867 in → 209 out (205 reasoning)
3bbbde0
x-ai/grok-4.3:medium__json_schema PASS 100% 934 in → 264 out (256 reasoning)
b56161b
JSON Schema Structured Output
(raw) {"hash":"b56161b"}
x-ai/grok-4.3:none PASS 100% 873 in → 5 out (0 reasoning)
1387d1f
x-ai/grok-4.3:none__json_schema PASS 100% 937 in → 9 out (0 reasoning)
d047e3e
JSON Schema Structured Output
(raw) {"hash":"d047e3e"}
x-ai/grok-4.3:xhigh PASS 100% 876 in → 173 out (170 reasoning)
89faaee
x-ai/grok-4.3:xhigh__json_schema PASS 100% 937 in → 280 out (271 reasoning)
ac0ec2b
JSON Schema Structured Output
(raw) {"hash":"ac0ec2b"}
x-ai/grok-4.5:high PASS 100% 922 in → 110 out (107 reasoning)
fa103ad
x-ai/grok-4.5:high__json_schema PASS 100% 997 in → 183 out (172 reasoning)
2b8f4de
JSON Schema Structured Output
(raw) {"hash":"2b8f4de"}
x-ai/grok-4.5:low PASS 100% 927 in → 129 out (124 reasoning)
afb6d16
x-ai/grok-4.5:low__json_schema PASS 100% 978 in → 161 out (152 reasoning)
15bf9ef
JSON Schema Structured Output
(raw) {"hash":"15bf9ef"}
x-ai/grok-4.5:max PASS 100% 916 in → 123 out (118 reasoning)
293b6c3
x-ai/grok-4.5:max__json_schema PASS 100% 978 in → 220 out (211 reasoning)
31403c6
JSON Schema Structured Output
(raw) {"hash":"31403c6"}
x-ai/grok-4.5:medium PASS 100% 910 in → 124 out (119 reasoning)
71ca9b4
x-ai/grok-4.5:medium__json_schema PASS 100% 988 in → 198 out (188 reasoning)
4d1c074
JSON Schema Structured Output
(raw) {"hash":"4d1c074"}
x-ai/grok-4.5:xhigh PASS 100% 908 in → 122 out (118 reasoning)
2f644db
x-ai/grok-4.5:xhigh__json_schema PASS 100% 978 in → 215 out (205 reasoning)
4a18dc6
JSON Schema Structured Output
(raw) {"hash":"4a18dc6"}
z-ai/glm-4.7-flash:high PASS 100% 754 in → 456 out (413 reasoning)
7906628
z-ai/glm-4.7-flash:high__json_schema PASS 100% 752 in → 419 out (352 reasoning)
4c502b8
JSON Schema Structured Output
(raw) { "hash": "4c502b8" }
z-ai/glm-4.7-flash:low PASS 100% 760 in → 667 out (581 reasoning)
6dfb0e2
z-ai/glm-4.7-flash:medium PASS 100% 744 in → 590 out (582 reasoning)
b0a21f8
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 737 in → 548 out (436 reasoning)
0279835
JSON Schema Structured Output
(raw) { "hash": "0279835" }
z-ai/glm-4.7-flash:none PASS 100% 750 in → 5 out (0 reasoning)
61503d6
z-ai/glm-4.7-flash:none__json_schema PASS 100% 742 in → 16 out (0 reasoning)
a8ee1ab
JSON Schema Structured Output
(raw) { "hash": "a8ee1ab" }
z-ai/glm-4.7-flash:xhigh PASS 100% 744 in → 310 out (275 reasoning)
69aed5d
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 753 in → 406 out (352 reasoning)
b26aea7
JSON Schema Structured Output
(raw) { "hash": "b26aea7" }
z-ai/glm-5.2:high PASS 100% 772 in → 116 out (107 reasoning)
5eed259
z-ai/glm-5.2:high__json_schema PASS 100% 765 in → 229 out (172 reasoning)
d5de9e6
JSON Schema Structured Output
(raw) { "hash": "d5de9e6" }
z-ai/glm-5.2:low PASS 100% 764 in → 78 out (73 reasoning)
78bc744
z-ai/glm-5.2:low__json_schema PASS 100% 744 in → 49 out (36 reasoning)
aaf852a
JSON Schema Structured Output
(raw) { "hash": "aaf852a" }
z-ai/glm-5.2:medium PASS 100% 768 in → 249 out (239 reasoning)
98b0a97
z-ai/glm-5.2:medium__json_schema PASS 100% 753 in → 77 out (64 reasoning)
c19add4
JSON Schema Structured Output
(raw) { "hash": "c19add4" }
z-ai/glm-5.2:none PASS 100% 769 in → 7 out (0 reasoning)
a7e8d4c
z-ai/glm-5.2:none__json_schema PASS 100% 767 in → 14 out (0 reasoning)
b551b76
JSON Schema Structured Output
(raw) { "hash": "b551b76" }
z-ai/glm-5.2:xhigh PASS 100% 758 in → 189 out (169 reasoning)
bd2ed31
z-ai/glm-5.2:xhigh__json_schema PASS 100% 945 in → 400 out (385 reasoning)
8cd145f
JSON Schema Structured Output
(raw) { "hash": "8cd145f" }
anthropic/claude-fable-5:max FAIL 0% 956 in → 3 out (0 reasoning)
(empty output)
Failure: Expected short hash 41e8df5, got
anthropic/claude-fable-5:xhigh FAIL 0% 955 in → 1 out (0 reasoning)
(empty output)
Failure: Expected short hash 4cf38d3, got
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 4ec26{ "hash": "4ec26b0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: 4bb78143b4713c50054123c9442429d674ace40b
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 428658d
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 8 (char 7)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 8 (char 7)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-20b:low__json_schema FAIL 0% 779 in → 359 out (0 reasoning)
The……
JSON Schema Structured Output
(raw) { "hash" : "The……" }
Failure: Expected short hash 40c58f2, got The……
openai/gpt-oss-20b:medium__json_schema FAIL 0% 741 in → 142 out (104 reasoning)
5a8fe09c0ba107a68ba61ecddb833bdb62bf3dc5
JSON Schema Structured Output
(raw) { "hash": "5a8fe09c0ba107a68ba61ecddb833bdb62bf3dc5" }
Failure: Expected short hash 5a8fe09, got 5a8fe09c0ba107a68ba61ecddb833bdb62bf3dc5
Invalid JSON. Output: 64951dc
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 6 (char 5)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 6 (char 5)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-036ad7f6-a785-934b-9f25-1eac5f516e3a","request_id":"036ad7f6-a785-934b-9f25-1eac5f516e3a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-41d77a27-7470-9310-a4fe-f08e1a52df1c","request_id":"41d77a27-7470-9310-a4fe-f08e1a52df1c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-22bc41f7-1111-9a6f-afa2-ef250d03d167","request_id":"22bc41f7-1111-9a6f-afa2-ef250d03d167"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-862bde53-f47c-94c3-b7c1-243032f899db","request_id":"862bde53-f47c-94c3-b7c1-243032f899db"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e5d97f8f-d29d-91af-9b1c-a818a513b1da","request_id":"e5d97f8f-d29d-91af-9b1c-a818a513b1da"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ca2d1538-926f-9933-8d17-b009b5a3bd85"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4e6ff96f-ff57-95ba-aa8b-c93b9e9d4ae2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-051b192f-ac58-996d-80d7-a528d89efbeb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8c619575-c6c7-95a4-88fc-5cccd19de671"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dfe520ce-a702-94c0-92cf-12d12ab97979"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7a62fda6-b3e1-998a-8435-6f504737e4f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6b91a6f1-9176-9aca-b038-5321a32b0d3e","request_id":"6b91a6f1-9176-9aca-b038-5321a32b0d3e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a0448ffd-d1a6-9202-a68e-4fd05131c2ca","request_id":"a0448ffd-d1a6-9202-a68e-4fd05131c2ca"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-41693ae0-f629-9da2-9832-342c469237f7","request_id":"41693ae0-f629-9da2-9832-342c469237f7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-642e9d48-fbab-977f-b604-6d83868ffcb2","request_id":"642e9d48-fbab-977f-b604-6d83868ffcb2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43f694cc-f8c3-9d5b-b90a-35bf3169a040","request_id":"43f694cc-f8c3-9d5b-b90a-35bf3169a040"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-722dd045-47ab-98e4-8b03-97c3cc167132","request_id":"722dd045-47ab-98e4-8b03-97c3cc167132"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-73f2247b-0fda-9be9-87c4-cc6bff39b117","request_id":"73f2247b-0fda-9be9-87c4-cc6bff39b117"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-71a020db-85bf-9ab3-8e05-b252508ce3eb","request_id":"71a020db-85bf-9ab3-8e05-b252508ce3eb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-771082eb-8e52-9744-a62d-11f44c9a97c3","request_id":"771082eb-8e52-9744-a62d-11f44c9a97c3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7df4964d-edb7-9857-a6d6-701cbf055dd2","request_id":"7df4964d-edb7-9857-a6d6-701cbf055dd2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: 5df6a7a{ "hash": "5df6a7a" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: 447178f
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 7 (char 6)
Invalid JSON. Output: ee83bb5
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)