Show commit full hash
Tests ability to extract full commit hash from git show. Evaluates SHA-extraction from commit display.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'initial' > seed.txt
  5. 05 git add seed.txt
  6. 06 git commit -m 'Seed commit'
  7. 07 echo 'second' > seed.txt
  8. 08 git add seed.txt
  9. 09 git commit -m 'Second commit'
Prompt
Using git show -s --format=%H, what is the full SHA hash of the commit with message 'Second commit'? Output ONLY the full hash, nothing else.
Expected
full hash for commit message 'Second commit'

Scoped model quality, cost, API time, and token usage for git_show/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 251 in → 26 out (0 reasoning)
a58bb9c1788ab1775394fab2611873b9a2354532
anthropic/claude-fable-5:high__json_schema PASS 100% 470 in → 41 out (0 reasoning)
07ee983fd3ca888f4e5ea3e7f55e27aaa6b3c425
JSON Schema Structured Output
(raw) {"hash": "07ee983fd3ca888f4e5ea3e7f55e27aaa6b3c425"}
anthropic/claude-fable-5:low PASS 100% 258 in → 32 out (0 reasoning)
8fb130bd2b85ccf1a4d32550605b7fdfd0a2a248
anthropic/claude-fable-5:low__json_schema PASS 100% 465 in → 37 out (0 reasoning)
e6f7ce60a7187a88689ec059ffdaec030fed1edc
JSON Schema Structured Output
(raw) {"hash": "e6f7ce60a7187a88689ec059ffdaec030fed1edc"}
anthropic/claude-fable-5:max PASS 100% 252 in → 92 out (36 reasoning)
3edca48435415f4697cb8d57d5de8b34a472a411
anthropic/claude-fable-5:max__json_schema PASS 100% 463 in → 100 out (32 reasoning)
ff9ee62a485ce413972487c1c32fb57270e736a9
JSON Schema Structured Output
(raw) {"hash": "ff9ee62a485ce413972487c1c32fb57270e736a9"}
anthropic/claude-fable-5:medium PASS 100% 251 in → 26 out (0 reasoning)
ee3b587be32b9529020a3f5bb866687a947327f4
anthropic/claude-fable-5:medium__json_schema PASS 100% 462 in → 33 out (0 reasoning)
5ff87a164243562cfc62f63d9440ab786320d117
JSON Schema Structured Output
(raw) {"hash": "5ff87a164243562cfc62f63d9440ab786320d117"}
anthropic/claude-fable-5:xhigh PASS 100% 252 in → 38 out (12 reasoning)
5614251c7c864b3e7249882daa4dea005c9b489b
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 464 in → 35 out (0 reasoning)
939fd871cc3df80790d5c20db3862ed8d9a44752
JSON Schema Structured Output
(raw) {"hash": "939fd871cc3df80790d5c20db3862ed8d9a44752"}
anthropic/claude-haiku-4.5:high PASS 100% 238 in → 170 out (112 reasoning)
98f375cfa2b8d1c7a729a98e461306455d61ae0e
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 396 in → 226 out (153 reasoning)
c9dd43138d40a0cde2791c72f29299399d132607
JSON Schema Structured Output
(raw) {"hash": "c9dd43138d40a0cde2791c72f29299399d132607"}
anthropic/claude-haiku-4.5:low PASS 100% 239 in → 180 out (107 reasoning)
99f7f2ba58dc64c02e302d00827c73f6fe3ce8f8
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 397 in → 197 out (135 reasoning)
4400393b9a6caad9b549833d54f156bcf938cd15
JSON Schema Structured Output
(raw) {"hash": "4400393b9a6caad9b549833d54f156bcf938cd15"}
anthropic/claude-haiku-4.5:medium PASS 100% 236 in → 132 out (90 reasoning)
329f9e2970db1a86d1ae38912056a9518e0aa861
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 397 in → 261 out (214 reasoning)
7389fb27831182fc3e9d6b254fe5bfddf2170124
JSON Schema Structured Output
(raw) {"hash": "7389fb27831182fc3e9d6b254fe5bfddf2170124"}
anthropic/claude-haiku-4.5:none PASS 100% 207 in → 27 out (0 reasoning)
e8fd887351517e4ebb1b6213a73ea063f43aa3f2
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 366 in → 30 out (0 reasoning)
eaacecdee65ef737de40889b7ca1ec7ab7ca32cd
JSON Schema Structured Output
(raw) {"hash": "eaacecdee65ef737de40889b7ca1ec7ab7ca32cd"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 240 in → 185 out (106 reasoning)
d3d93651b1d6b43bf7843e10c2e72d1e7355a68a
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 397 in → 259 out (171 reasoning)
782eab0ff0c335775b1740afeb146ae9bae039c3
JSON Schema Structured Output
(raw) {"hash": "782eab0ff0c335775b1740afeb146ae9bae039c3"}
anthropic/claude-opus-4.6:high PASS 100% 208 in → 28 out (0 reasoning)
c3ad66bae68b29b959c0c7d13353a79ff64820c0
anthropic/claude-opus-4.6:high__json_schema PASS 100% 366 in → 27 out (0 reasoning)
1f6698557cc14ab85827bd3b661287457dd8baef
JSON Schema Structured Output
(raw) {"hash":"1f6698557cc14ab85827bd3b661287457dd8baef"}
anthropic/claude-opus-4.6:low PASS 100% 208 in → 28 out (0 reasoning)
0a09ab511b42ad336a8c2f9efdd8635816b28ac5
anthropic/claude-opus-4.6:low__json_schema PASS 100% 370 in → 31 out (0 reasoning)
2c50ba07e97c71c0b363cd9ee010197056bba7d7
JSON Schema Structured Output
(raw) {"hash":"2c50ba07e97c71c0b363cd9ee010197056bba7d7"}
anthropic/claude-opus-4.6:max PASS 100% 207 in → 27 out (0 reasoning)
0057c9d4e89934a60af00fd6cc6b74af885905cf
anthropic/claude-opus-4.6:max__json_schema PASS 100% 368 in → 30 out (0 reasoning)
740fe596cfcaa6cf3e6f9616447459bc37d3c126
JSON Schema Structured Output
(raw) {"hash":"740fe596cfcaa6cf3e6f9616447459bc37d3c126"}
anthropic/claude-opus-4.6:medium PASS 100% 206 in → 26 out (0 reasoning)
92fec30fa91bb308c44cf57f567073f0830c2da0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 370 in → 32 out (0 reasoning)
b9c2d194af7e1b277b7ab404085bf4ef3fe4cc95
JSON Schema Structured Output
(raw) {"hash":"b9c2d194af7e1b277b7ab404085bf4ef3fe4cc95"}
anthropic/claude-opus-4.6:none PASS 100% 206 in → 26 out (0 reasoning)
d55637e5956e3c440bb6046a30c9d341156a7352
anthropic/claude-opus-4.6:none__json_schema PASS 100% 366 in → 27 out (0 reasoning)
694d942825cdd5ddaba68703e34a897d663c7781
JSON Schema Structured Output
(raw) {"hash":"694d942825cdd5ddaba68703e34a897d663c7781"}
anthropic/claude-opus-4.6:xhigh PASS 100% 207 in → 27 out (0 reasoning)
721aff8552d72fe187d86af65312ed7403a3a6a4
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 368 in → 29 out (0 reasoning)
7a08577f341bf558fb82e6847ab7b572eea4a888
JSON Schema Structured Output
(raw) {"hash":"7a08577f341bf558fb82e6847ab7b572eea4a888"}
anthropic/claude-opus-4.7:high PASS 100% 258 in → 30 out (0 reasoning)
cc20b84e97d32913f94f01dc628bb0db720b88b7
anthropic/claude-opus-4.7:high__json_schema PASS 100% 472 in → 40 out (0 reasoning)
8543d44e8ea36ddd108e7ee944c69e90d9caae7c
JSON Schema Structured Output
(raw) {"hash":"8543d44e8ea36ddd108e7ee944c69e90d9caae7c"}
anthropic/claude-opus-4.7:low PASS 100% 258 in → 30 out (0 reasoning)
c36a56003e6fb469bca2052e913e59e08c1afdbe
anthropic/claude-opus-4.7:low__json_schema PASS 100% 472 in → 40 out (0 reasoning)
995f5f6cc9359e34f60b3583a16d41dbc5feacd8
JSON Schema Structured Output
(raw) {"hash":"995f5f6cc9359e34f60b3583a16d41dbc5feacd8"}
anthropic/claude-opus-4.7:max PASS 100% 259 in → 31 out (0 reasoning)
7aef0505ad6dfeb45785b410cc6069461c9632d9
anthropic/claude-opus-4.7:max__json_schema PASS 100% 468 in → 37 out (0 reasoning)
61830d20063f3eb928ef52658b06547d265bd3d3
JSON Schema Structured Output
(raw) {"hash":"61830d20063f3eb928ef52658b06547d265bd3d3"}
anthropic/claude-opus-4.7:medium PASS 100% 257 in → 29 out (0 reasoning)
9a6104e040ca46700ac5f6174b3f73595005d144
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 468 in → 36 out (0 reasoning)
3294257df421e49d440aabfb4467cfb21581c6b7
JSON Schema Structured Output
(raw) {"hash":"3294257df421e49d440aabfb4467cfb21581c6b7"}
anthropic/claude-opus-4.7:none PASS 100% 262 in → 34 out (0 reasoning)
bbfedef7efc582a87b6bace28f3bd9bba6ce1f9c
anthropic/claude-opus-4.7:none__json_schema PASS 100% 471 in → 39 out (0 reasoning)
8dee4a98c30e53e739b5682c898a56ff5be2ac34
JSON Schema Structured Output
(raw) {"hash":"8dee4a98c30e53e739b5682c898a56ff5be2ac34"}
anthropic/claude-opus-4.7:xhigh PASS 100% 258 in → 30 out (0 reasoning)
cbfbf874fdda746e9c5d894b84f8c160eba525be
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 468 in → 36 out (0 reasoning)
3f750684723f730d2a38901269c0f34472b0d1f1
JSON Schema Structured Output
(raw) {"hash":"3f750684723f730d2a38901269c0f34472b0d1f1"}
anthropic/claude-opus-4.8:high PASS 100% 258 in → 33 out (0 reasoning)
b6b616d82bbd1e7b426f0b60db9e6ee0dd4bd885
anthropic/claude-opus-4.8:high__json_schema PASS 100% 466 in → 37 out (0 reasoning)
129e7e81c1dae1734c1609f84d55832f37e4d6a4
JSON Schema Structured Output
(raw) {"hash":"129e7e81c1dae1734c1609f84d55832f37e4d6a4"}
anthropic/claude-opus-4.8:low PASS 100% 255 in → 29 out (0 reasoning)
2e536c0cf7ef93332b05b33d9853acb1af85bd86
anthropic/claude-opus-4.8:low__json_schema PASS 100% 460 in → 32 out (0 reasoning)
f09e13edb4530704036d7e428619619640bc6d91
JSON Schema Structured Output
(raw) {"hash":"f09e13edb4530704036d7e428619619640bc6d91"}
anthropic/claude-opus-4.8:max PASS 100% 254 in → 201 out (43 reasoning)
42c313dae3768749b62931c5fcacaf425c4ee2c1
anthropic/claude-opus-4.8:max__json_schema PASS 100% 463 in → 101 out (26 reasoning)
e256ff839c29a30523271ebb1c25060c78e4fc93
JSON Schema Structured Output
(raw) {"hash": "e256ff839c29a30523271ebb1c25060c78e4fc93"}
anthropic/claude-opus-4.8:medium PASS 100% 255 in → 30 out (0 reasoning)
f30437b0d0f33ea5e33a1b782bc61df58c80f65f
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 465 in → 37 out (0 reasoning)
ae77dacd99b31e8768d611bf1c8adcd87df923ac
JSON Schema Structured Output
(raw) {"hash":"ae77dacd99b31e8768d611bf1c8adcd87df923ac"}
anthropic/claude-opus-4.8:none PASS 100% 256 in → 31 out (0 reasoning)
baa2c18dbb1f9a6c6796e07f6c7327cd296759bb
anthropic/claude-opus-4.8:none__json_schema PASS 100% 465 in → 36 out (0 reasoning)
daf9edd8f4b01a171c89c602b6d1c3439feabe57
JSON Schema Structured Output
(raw) {"hash":"daf9edd8f4b01a171c89c602b6d1c3439feabe57"}
anthropic/claude-opus-4.8:xhigh PASS 100% 259 in → 34 out (0 reasoning)
d330a87a4a9b3a88e10cf8bb568bf7f5c8ec41b8
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 461 in → 32 out (0 reasoning)
6de652c900447d4faa63f5917384831bf83402f7
JSON Schema Structured Output
(raw) {"hash":"6de652c900447d4faa63f5917384831bf83402f7"}
anthropic/claude-sonnet-4.6:high PASS 100% 208 in → 28 out (0 reasoning)
ffa2c80321f2fa1067e7d89655aa5529b54fbc8b
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 363 in → 25 out (0 reasoning)
af79c89a7785477bacc530fe438fc500245856fc
JSON Schema Structured Output
(raw) {"hash":"af79c89a7785477bacc530fe438fc500245856fc"}
anthropic/claude-sonnet-4.6:low PASS 100% 206 in → 26 out (0 reasoning)
08a16f9fd00670cca335b950213ff25d707b62e0
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 366 in → 27 out (0 reasoning)
0de01051440e77bc03110ae3b42ca627564df55b
JSON Schema Structured Output
(raw) {"hash":"0de01051440e77bc03110ae3b42ca627564df55b"}
anthropic/claude-sonnet-4.6:max PASS 100% 210 in → 112 out (55 reasoning)
f4ebe1b1e45d065f8c9b29691fabb27ca65c0d61
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 370 in → 31 out (0 reasoning)
10079acbbcab21ade4aece7b50e4678c87adf4fb
JSON Schema Structured Output
(raw) {"hash":"10079acbbcab21ade4aece7b50e4678c87adf4fb"}
anthropic/claude-sonnet-4.6:medium PASS 100% 206 in → 26 out (0 reasoning)
6b88bda3a256853813b0dcdca19372f25037283c
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 372 in → 34 out (0 reasoning)
a563956f7d6b4baed2fffb0db1bb0b9dc28e1071
JSON Schema Structured Output
(raw) {"hash":"a563956f7d6b4baed2fffb0db1bb0b9dc28e1071"}
anthropic/claude-sonnet-4.6:none PASS 100% 208 in → 28 out (0 reasoning)
eb2c7f8e18f3f79d775883504f689429228e6b7d
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 371 in → 32 out (0 reasoning)
4cda66d1f4a417070b201e1447e1481ee74f8ec3
JSON Schema Structured Output
(raw) {"hash":"4cda66d1f4a417070b201e1447e1481ee74f8ec3"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 206 in → 26 out (0 reasoning)
64cd8b75af4aef5bc62fb58ea68bc51eb416af63
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 370 in → 32 out (0 reasoning)
d8b3508878a650f04a4f374f358d1feafbce28e1
JSON Schema Structured Output
(raw) {"hash":"d8b3508878a650f04a4f374f358d1feafbce28e1"}
anthropic/claude-sonnet-5:high PASS 100% 254 in → 28 out (0 reasoning)
816b42a0e107c3ceeae05ab143ddb50e0276606c
anthropic/claude-sonnet-5:high__json_schema PASS 100% 462 in → 33 out (0 reasoning)
415da67ace86586f45cf9c4435395e4272ca8383
JSON Schema Structured Output
(raw) {"hash": "415da67ace86586f45cf9c4435395e4272ca8383"}
anthropic/claude-sonnet-5:low PASS 100% 255 in → 29 out (0 reasoning)
155a139790a7f172d0e7d6bedbafdb1833bb40e1
anthropic/claude-sonnet-5:low__json_schema PASS 100% 465 in → 37 out (0 reasoning)
b855a5c60c0a925dc0341b2dc823139a16d11ce2
JSON Schema Structured Output
(raw) {"hash": "b855a5c60c0a925dc0341b2dc823139a16d11ce2"}
anthropic/claude-sonnet-5:medium PASS 100% 255 in → 29 out (0 reasoning)
81758bc6db5b34332c046dafefc6ea292d3ecf5a
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 466 in → 37 out (0 reasoning)
5f690ff6ec715bd47e76917c09396e2aa65ef50f
JSON Schema Structured Output
(raw) {"hash": "5f690ff6ec715bd47e76917c09396e2aa65ef50f"}
anthropic/claude-sonnet-5:none PASS 100% 256 in → 31 out (0 reasoning)
c9bbfdc1e01a43d13a443892efb1d8910ac1a64e
anthropic/claude-sonnet-5:none__json_schema PASS 100% 466 in → 36 out (0 reasoning)
0fa03f4c751d28d93a67f83bb66e89d9187572e1
JSON Schema Structured Output
(raw) {"hash": "0fa03f4c751d28d93a67f83bb66e89d9187572e1"}
anthropic/claude-sonnet-5:xhigh PASS 100% 255 in → 353 out (74 reasoning)
5df0ae7321fe48bd1269253bc3faab1d80c666d6
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 466 in → 37 out (0 reasoning)
687187ce9c02e1138a7f3d75ba12fa2e042ba721
JSON Schema Structured Output
(raw) {"hash": "687187ce9c02e1138a7f3d75ba12fa2e042ba721"}
arcee-ai/trinity-large-thinking:high PASS 100% 177 in → 363 out (342 reasoning)
fdffdb85bcae97021d551ffd6adec526d0b3a18a
arcee-ai/trinity-large-thinking:low PASS 100% 178 in → 306 out (282 reasoning)
59e8fb6564e8953363e50a3779353302d09df809
arcee-ai/trinity-large-thinking:medium PASS 100% 178 in → 321 out (297 reasoning)
5c91611ccaac70914f3d5469cc5c639acf9507ae
arcee-ai/trinity-large-thinking:xhigh PASS 100% 179 in → 478 out (453 reasoning)
34af58431bcc035d1963d50a6dbc55c029d07f3b
arcee-ai/trinity-mini:high PASS 100% 178 in → 261 out (242 reasoning)
ab70f21db2a5fdbd208f1766474833409c8ba5ec
arcee-ai/trinity-mini:high__json_schema PASS 100% 182 in → 307 out (262 reasoning)
f5c2a3781d61ea363b2e7af4b6858cf6971f2249
JSON Schema Structured Output
(raw) { "hash": "f5c2a3781d61ea363b2e7af4b6858cf6971f2249" }
arcee-ai/trinity-mini:low PASS 100% 182 in → 296 out (276 reasoning)
52bb7c23b5f887ac5a7f176a1eae61bf96262d8a
arcee-ai/trinity-mini:low__json_schema PASS 100% 181 in → 309 out (278 reasoning)
15d5b0e70e18a95419026d598e83b70f12141f26
JSON Schema Structured Output
(raw) { "hash":"15d5b0e70e18a95419026d598e83b70f12141f26" }
arcee-ai/trinity-mini:medium PASS 100% 181 in → 260 out (211 reasoning)
60c23806db4eafa7dca2fa5343c7f7029361e6f2
arcee-ai/trinity-mini:medium__json_schema PASS 100% 183 in → 306 out (263 reasoning)
9a4aca6c5d4bfad3275a652b9d5f6dab82d62005
JSON Schema Structured Output
(raw) { "hash": "9a4aca6c5d4bfad3275a652b9d5f6dab82d62005" }
arcee-ai/trinity-mini:xhigh PASS 100% 173 in → 295 out (285 reasoning)
ee0fda729542c495393385af87adeba72bcdfea9
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 180 in → 294 out (235 reasoning)
007d8558c3cbd856878a01f72b9de73a9fc853f2
JSON Schema Structured Output
(raw) { "hash": "007d8558c3cbd856878a01f72b9de73a9fc853f2" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 187 in → 101 out (73 reasoning)
fa0f79fbb9db325e5f2a6e883e23e99905c0db96
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 260 in → 87 out (47 reasoning)
580d41dba0c97d07922291a5c0659e75d0f9e669
JSON Schema Structured Output
(raw) {"hash":"580d41dba0c97d07922291a5c0659e75d0f9e669"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 175 in → 68 out (46 reasoning)
de7bea239773da0614826a4959bcdda266cbcaaa
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 178 in → 119 out (80 reasoning)
0f901f32937df10c63c5969a86ac7664265827b0
JSON Schema Structured Output
(raw) { "hash": "0f901f32937df10c63c5969a86ac7664265827b0" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 278 in → 95 out (63 reasoning)
6ba9aa7aca319df47f2f5dc8ba7c1057681355d1
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 273 in → 260 out (161 reasoning)
cfa89ceb7de515b213a94e85b1b6f6b7378157b5
JSON Schema Structured Output
(raw) { "hash": "cfa89ceb7de515b213a94e85b1b6f6b7378157b5" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 181 in → 87 out (47 reasoning)
f1bf8e25c11931aa44a4adc48c9be3e9d3256733
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 179 in → 113 out (76 reasoning)
2a25dd430275dcfc2a428807d47c0a1ed232c1fe
JSON Schema Structured Output
(raw) {"hash": "2a25dd430275dcfc2a428807d47c0a1ed232c1fe"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 177 in → 22 out (0 reasoning)
d0a46deb99678da0de07a41a66ccaf942e824f48
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 176 in → 30 out (0 reasoning)
57ad466d5f7fd098c58882597356c91c441bd078
JSON Schema Structured Output
(raw) { "hash": "57ad466d5f7fd098c58882597356c91c441bd078" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 258 in → 80 out (47 reasoning)
7bc65669657ff8e8f2d7e8514816d072833f1125
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 262 in → 74 out (35 reasoning)
cf9535e285596330431f87a1cf7eee0b2062d63b
JSON Schema Structured Output
(raw) { "hash": "cf9535e285596330431f87a1cf7eee0b2062d63b" }
deepseek/deepseek-v4-flash:high PASS 100% 181 in → 91 out (52 reasoning)
dd4ce50cbd6134d2d240caea08ec7f1e0f563f2c
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 183 in → 160 out (125 reasoning)
26cf8b1ff9c098790b04597b7f5ae2cdfbee32b3
JSON Schema Structured Output
(raw) { "hash": "26cf8b1ff9c098790b04597b7f5ae2cdfbee32b3" }
deepseek/deepseek-v4-flash:low PASS 100% 180 in → 121 out (85 reasoning)
55f5774969b4507a9bc2e5b1303c3a686819449f
deepseek/deepseek-v4-flash:medium PASS 100% 178 in → 114 out (79 reasoning)
16339cd24e236e6d3627f128c83996913a909c3a
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 178 in → 155 out (126 reasoning)
1766061d4f900ed4cf0900bec65a959705615ae2
JSON Schema Structured Output
(raw) { "hash": "1766061d4f900ed4cf0900bec65a959705615ae2"}
deepseek/deepseek-v4-flash:none PASS 100% 177 in → 22 out (0 reasoning)
75f0d1d54451544354977d02047845efc9165e94
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 176 in → 28 out (0 reasoning)
18628346852ff8842e285102b98ae2847c1441d3
JSON Schema Structured Output
(raw) { "hash": "18628346852ff8842e285102b98ae2847c1441d3" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 257 in → 127 out (102 reasoning)
c77ea08ef60deec160110c7d713d9cc61ee05f0c
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 256 in → 190 out (156 reasoning)
f16b2ad36a58434ff41fde14f1de50470364db41
JSON Schema Structured Output
(raw) { "hash": "f16b2ad36a58434ff41fde14f1de50470364db41" }
deepseek/deepseek-v4-pro:high PASS 100% 179 in → 165 out (139 reasoning)
c324e3638074de947d5f8d03f0948bc8c899cff8
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 180 in → 35 out (0 reasoning)
bf4b465f02bdc65e7705112a514bb6d988e4f09d
JSON Schema Structured Output
(raw) { "hash": "bf4b465f02bdc65e7705112a514bb6d988e4f09d" }
deepseek/deepseek-v4-pro:low PASS 100% 176 in → 110 out (87 reasoning)
e377276725ddd55a1d05ab5f591426d06cb2fe69
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 363 in → 149 out (120 reasoning)
a8194461a9abdb45be9f49fadde79eb6d304bbbb
JSON Schema Structured Output
(raw) {"hash":"a8194461a9abdb45be9f49fadde79eb6d304bbbb"}
deepseek/deepseek-v4-pro:medium PASS 100% 178 in → 157 out (132 reasoning)
fe5fde8bb8354f894d9ce7bfbe775ef6f371ddbb
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 182 in → 94 out (61 reasoning)
5e3cc3f05d92de8b514e4775bbabafb719a01356
JSON Schema Structured Output
(raw) { "hash": "5e3cc3f05d92de8b514e4775bbabafb719a01356" }
deepseek/deepseek-v4-pro:none PASS 100% 178 in → 22 out (0 reasoning)
37ce06d82455dae1acc8a5844ed46c3c71530044
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 361 in → 26 out (0 reasoning)
8afae6859fa2a523647049623be7ede6d938f608
JSON Schema Structured Output
(raw) {"hash": "8afae6859fa2a523647049623be7ede6d938f608"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 255 in → 147 out (124 reasoning)
b83d82ef350970b5de861159c11fc4560cb0f321
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 260 in → 151 out (118 reasoning)
43551fdb76ac8d824e8ef6b655a4af1976750fdc
JSON Schema Structured Output
(raw) { "hash": "43551fdb76ac8d824e8ef6b655a4af1976750fdc" }
google/gemini-3-flash-preview:high PASS 100% 206 in → 205 out (172 reasoning)
321ff9c6c8b1a6efe7bb7ab81508fc5055fdc1e0
google/gemini-3-flash-preview:high__json_schema PASS 100% 207 in → 402 out (364 reasoning)
ccbe9c53424459d1f52fbaad8d978fbf8757de92
JSON Schema Structured Output
(raw) {"hash":"ccbe9c53424459d1f52fbaad8d978fbf8757de92"}
google/gemini-3-flash-preview:low PASS 100% 210 in → 181 out (144 reasoning)
33388bc8635e7de4419a36857346509986cfb33d
google/gemini-3-flash-preview:low__json_schema PASS 100% 260 in → 36 out (0 reasoning)
37f39d3aaaea4060f70ffc6853cab9ef4c365fa3
JSON Schema Structured Output
(raw) {"hash":"37f39d3aaaea4060f70ffc6853cab9ef4c365fa3"}
google/gemini-3-flash-preview:medium PASS 100% 209 in → 306 out (270 reasoning)
1a51014b13bc2a1738b231a4bcee3be183e30e58
google/gemini-3-flash-preview:medium__json_schema PASS 100% 212 in → 409 out (366 reasoning)
b6210f6a15004328d35f9f402bde7930f51159c6
JSON Schema Structured Output
(raw) {"hash":"b6210f6a15004328d35f9f402bde7930f51159c6"}
google/gemini-3-flash-preview:xhigh PASS 100% 209 in → 189 out (152 reasoning)
f7b6c6a98b9080b45501092702d168d0320cbdba
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 264 in → 396 out (356 reasoning)
1550135fc4ae8585276661cf1a2c9a4f2523cbe0
JSON Schema Structured Output
(raw) {"hash":"1550135fc4ae8585276661cf1a2c9a4f2523cbe0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 207 in → 424 out (390 reasoning)
698c78ee6fda3f97b7b4de416fa2a990b239cdd8
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 261 in → 374 out (335 reasoning)
d73b44970eb97d812678ab90e42eed79502fc1fd
JSON Schema Structured Output
(raw) {"hash": "d73b44970eb97d812678ab90e42eed79502fc1fd"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 205 in → 154 out (122 reasoning)
0ecbb7dec1bdd4060e5d67a2ceb73f45e76777eb
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 261 in → 151 out (112 reasoning)
c13f2a20e01ac3c2ec458c257efafa74d7b1fe0d
JSON Schema Structured Output
(raw) {"hash": "c13f2a20e01ac3c2ec458c257efafa74d7b1fe0d"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 207 in → 205 out (171 reasoning)
0c0cdef183b21a36e07d40a4ab6cf4c4fee196a2
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 212 in → 360 out (312 reasoning)
3ef399c08c47c1b0b5d72a361582a46eb6c8f06f
JSON Schema Structured Output
(raw) { "hash": "3ef399c08c47c1b0b5d72a361582a46eb6c8f06f" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 210 in → 458 out (421 reasoning)
4f82fd42be1179e3448f4459e6662c3e6606c5da
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 266 in → 515 out (471 reasoning)
c2e5387e1507d706a7ac68f7f6d88715c3b6e225
JSON Schema Structured Output
(raw) {"hash": "c2e5387e1507d706a7ac68f7f6d88715c3b6e225"}
google/gemini-3.1-pro-preview:high PASS 100% 209 in → 490 out (454 reasoning)
773c52623e5c1ea6f7d03c46dd73be7ac21b6366
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 262 in → 420 out (383 reasoning)
5dc5d871f906b241daec7c77bf56796f266afe4b
JSON Schema Structured Output
(raw) {"hash":"5dc5d871f906b241daec7c77bf56796f266afe4b"}
google/gemini-3.1-pro-preview:low PASS 100% 208 in → 245 out (210 reasoning)
ccd056311bf2e1598fae16c56db9365f625a649d
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 260 in → 158 out (121 reasoning)
e03f355daacdc2a85ce56f5e76dff15356fdcc40
JSON Schema Structured Output
(raw) {"hash":"e03f355daacdc2a85ce56f5e76dff15356fdcc40"}
google/gemini-3.1-pro-preview:medium PASS 100% 211 in → 269 out (231 reasoning)
01f8002090578dc2f4ab2a8458d4a28534e53b38
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 264 in → 376 out (336 reasoning)
53689183555c0064f35701c55cfef221923dfbdf
JSON Schema Structured Output
(raw) {"hash":"53689183555c0064f35701c55cfef221923dfbdf"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 209 in → 366 out (329 reasoning)
a80cb168a63b730ff6d10b9686f2403cc8973687
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 264 in → 275 out (234 reasoning)
e6a38a63ebdba12029f1f7497f2d5c6888b01c53
JSON Schema Structured Output
(raw) {"hash":"e6a38a63ebdba12029f1f7497f2d5c6888b01c53"}
google/gemini-3.5-flash-lite:high PASS 100% 212 in → 386 out (349 reasoning)
0f944f587f8ee099f6008e37e594a0ff7fc9155c
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 263 in → 395 out (356 reasoning)
992479ee26902a441fd399b0144d26fcfcb8cfde
JSON Schema Structured Output
(raw) {"hash": "992479ee26902a441fd399b0144d26fcfcb8cfde"}
google/gemini-3.5-flash-lite:low PASS 100% 212 in → 37 out (0 reasoning)
75a2ba2806bc9188d5866104e1e45cd580384382
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 266 in → 47 out (0 reasoning)
7e472fb9e9e4adc4104c071f0357c2c88b2798e9
JSON Schema Structured Output
(raw) { "hash": "7e472fb9e9e4adc4104c071f0357c2c88b2798e9" }
google/gemini-3.5-flash-lite:max PASS 100% 210 in → 451 out (416 reasoning)
00f403ead1e8097a8fc0e8a2e09cb1006b79ea6f
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 211 in → 431 out (390 reasoning)
fce8f51684a063eff9212d6023e5a7b73b8e0666
JSON Schema Structured Output
(raw) {"hash":"fce8f51684a063eff9212d6023e5a7b73b8e0666"}
google/gemini-3.5-flash-lite:medium PASS 100% 212 in → 353 out (316 reasoning)
1fec099c30be61c1907758c54ec9411f4180a670
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 265 in → 236 out (196 reasoning)
9d070b1cea081b2a3ad7d4d7c88a70c14275ef9a
JSON Schema Structured Output
(raw) {"hash":"9d070b1cea081b2a3ad7d4d7c88a70c14275ef9a"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 214 in → 392 out (352 reasoning)
05e5a26a371523e7b6e22942261b730426074801
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 266 in → 485 out (444 reasoning)
9d75d3d285a671b13c67a9a3f3a31a92caf513fa
JSON Schema Structured Output
(raw) {"hash":"9d75d3d285a671b13c67a9a3f3a31a92caf513fa"}
google/gemini-3.5-flash:high PASS 100% 209 in → 358 out (322 reasoning)
663c19a9600c367f8ff6854d03c15acd82968ee5
google/gemini-3.5-flash:high__json_schema PASS 100% 261 in → 324 out (287 reasoning)
faf304eb28e76bb7fea6b735002ae5d9238894b4
JSON Schema Structured Output
(raw) {"hash":"faf304eb28e76bb7fea6b735002ae5d9238894b4"}
google/gemini-3.5-flash:low PASS 100% 208 in → 119 out (84 reasoning)
8e76dfd2af769d70b799ce38cb0b9a0f45dc3701
google/gemini-3.5-flash:low__json_schema PASS 100% 263 in → 134 out (95 reasoning)
0d99b933158fd86d5e8fff03d74ecb2f420fc022
JSON Schema Structured Output
(raw) {"hash":"0d99b933158fd86d5e8fff03d74ecb2f420fc022"}
google/gemini-3.5-flash:medium PASS 100% 210 in → 409 out (372 reasoning)
78718710bb6120422945fd3516d744bd4789792a
google/gemini-3.5-flash:medium__json_schema PASS 100% 264 in → 384 out (338 reasoning)
824c2cd57d47e18bb0cea93712d76387416e101d
JSON Schema Structured Output
(raw) { "hash": "824c2cd57d47e18bb0cea93712d76387416e101d" }
google/gemini-3.5-flash:xhigh PASS 100% 205 in → 355 out (323 reasoning)
3fe86e089c302ca7ef86fbfc04f9cffc1e08ece4
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 266 in → 445 out (403 reasoning)
1896e1c2589d302e9b74a8fdc09c0f4e6dc043d6
JSON Schema Structured Output
(raw) {"hash":"1896e1c2589d302e9b74a8fdc09c0f4e6dc043d6"}
google/gemini-3.6-flash:high PASS 100% 212 in → 354 out (316 reasoning)
45814e7aa132264a6d1a30b1c9a2eb39d39e6750
google/gemini-3.6-flash:high__json_schema PASS 100% 264 in → 374 out (335 reasoning)
016757e83f2edb5edb4831a2905b879af06ddd38
JSON Schema Structured Output
(raw) {"hash":"016757e83f2edb5edb4831a2905b879af06ddd38"}
google/gemini-3.6-flash:low PASS 100% 209 in → 145 out (110 reasoning)
fde4c03b26c86f6eafa907d5ab938831304dfc04
google/gemini-3.6-flash:low__json_schema PASS 100% 209 in → 39 out (0 reasoning)
ea8bb58eb99368eac7a6d380da364a17f8c79a77
JSON Schema Structured Output
(raw) {"hash":"ea8bb58eb99368eac7a6d380da364a17f8c79a77"}
google/gemini-3.6-flash:max PASS 100% 209 in → 358 out (324 reasoning)
95f294553599e1bc4dcd6f81feadeb40d8bb36b4
google/gemini-3.6-flash:max__json_schema PASS 100% 209 in → 328 out (289 reasoning)
ef8b17ed7530e5264ae58014d46afa53d60f1b6a
JSON Schema Structured Output
(raw) {"hash":"ef8b17ed7530e5264ae58014d46afa53d60f1b6a"}
google/gemini-3.6-flash:medium PASS 100% 210 in → 202 out (167 reasoning)
0fcb52aed449a27dc1e12ac3e2b5ab38b0f48389
google/gemini-3.6-flash:medium__json_schema PASS 100% 213 in → 246 out (203 reasoning)
f4b7d7c0a85634dbf2951a4424557d7514e12395
JSON Schema Structured Output
(raw) {"hash":"f4b7d7c0a85634dbf2951a4424557d7514e12395"}
google/gemini-3.6-flash:xhigh PASS 100% 210 in → 375 out (338 reasoning)
b40b744c2b90b0a51fb7e9e1d07bd8c9cb188c5a
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 259 in → 311 out (276 reasoning)
fea8ce95abd462afdac99150b621105fdf73e2c4
JSON Schema Structured Output
(raw) {"hash":"fea8ce95abd462afdac99150b621105fdf73e2c4"}
google/gemma-4-26b-a4b-it:high PASS 100% 227 in → 354 out (192 reasoning)
54988e4573df37d0d57d7540e5d22567e00b52dd
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 227 in → 321 out (150 reasoning)
0249550b78401c2b492f931157ddc8d534c7698a
JSON Schema Structured Output
(raw) { "hash": "0249550b78401c2b492f931157ddc8d534c7698a" }
google/gemma-4-26b-a4b-it:low PASS 100% 221 in → 312 out (174 reasoning)
bd09eb9c4cb5e9c244af20ad2a8c7c7b76db5329
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 224 in → 391 out (206 reasoning)
b9b30b1bd7f281fd50e286cfee6d38392c4613a4
JSON Schema Structured Output
(raw) { "hash": "b9b30b1bd7f281fd50e286cfee6d38392c4613a4" }
google/gemma-4-26b-a4b-it:medium PASS 100% 225 in → 477 out (440 reasoning)
6c00cec6462a387546ada8210e1659d2501c8b22
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 225 in → 528 out (336 reasoning)
3e4245d085b33dd3adca758be17928f578b68d88
JSON Schema Structured Output
(raw) { "hash": "3e4245d085b33dd3adca758be17928f578b68d88" }
google/gemma-4-26b-a4b-it:none PASS 100% 222 in → 37 out (0 reasoning)
19e6aaf0436f05238e3849afe76502284870445f
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 224 in → 49 out (0 reasoning)
5895b0fd6ac5559a4e227381058d4a30128b9b68
JSON Schema Structured Output
(raw) { "hash": "5895b0fd6ac5559a4e227381058d4a30128b9b68" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 221 in → 307 out (173 reasoning)
ed1e2cca73fa9d24d1ac69b3b7d094ac6da65c85
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 223 in → 326 out (178 reasoning)
7de1e254477857007231cd2c3c7a25cb7b12ede1
JSON Schema Structured Output
(raw) { "hash": "7de1e254477857007231cd2c3c7a25cb7b12ede1" }
google/gemma-4-31b-it:high PASS 100% 220 in → 277 out (1 reasoning)
ec7d70fcbc3cbc1bbca95cb239468461c2f25f62
google/gemma-4-31b-it:high__json_schema PASS 100% 224 in → 255 out (134 reasoning)
0673e4ee040a4386dc8d207b9291d217cefa3066
JSON Schema Structured Output
(raw) { "hash": "0673e4ee040a4386dc8d207b9291d217cefa3066" }
google/gemma-4-31b-it:low PASS 100% 226 in → 296 out (180 reasoning)
33388bc8635e7de4419a36857346509986cfb33d
google/gemma-4-31b-it:low__json_schema PASS 100% 226 in → 378 out (208 reasoning)
e4309471627854bb620230d59ac78e5922017107
JSON Schema Structured Output
(raw) {"hash": "e4309471627854bb620230d59ac78e5922017107"}
google/gemma-4-31b-it:medium PASS 100% 228 in → 271 out (147 reasoning)
7158a8d363e1b4d933924b82ee1f76e73a624953
google/gemma-4-31b-it:medium__json_schema PASS 100% 226 in → 271 out (143 reasoning)
994506010a854008c979cbc92e65d72e130950af
JSON Schema Structured Output
(raw) { "hash": "994506010a854008c979cbc92e65d72e130950af" }
google/gemma-4-31b-it:none PASS 100% 217 in → 33 out (0 reasoning)
c7ebfb2240a7df1fc691df18cc14dadd1c45c2b1
google/gemma-4-31b-it:none__json_schema PASS 100% 221 in → 43 out (0 reasoning)
00ac8bcda071a1234977e8048cf60eaf9c7e2b60
JSON Schema Structured Output
(raw) { "hash": "00ac8bcda071a1234977e8048cf60eaf9c7e2b60" }
google/gemma-4-31b-it:xhigh PASS 100% 226 in → 276 out (160 reasoning)
3584ea295ecfb63a823470228483456547157020
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 227 in → 316 out (159 reasoning)
7bd2b36d8ae6015c9340469694291232498703e8
JSON Schema Structured Output
(raw) { "hash": "7bd2b36d8ae6015c9340469694291232498703e8" }
ibm-granite/granite-4.1-8b PASS 100% 196 in → 26 out
a61d1ef441538d5d2f7dfbe3d62d3f87075df911
ibm-granite/granite-4.1-8b__json_schema PASS 100% 191 in → 29 out
bf17ad54d54a464fe8557413fa98180fc23ef5a5
JSON Schema Structured Output
(raw) { "hash": "bf17ad54d54a464fe8557413fa98180fc23ef5a5" }
inclusionai/ling-2.6-flash PASS 100% 225 in → 36 out
954ddd681b740c40669f4dfcfd74073cd642156f
inclusionai/ling-2.6-flash__json_schema PASS 100% 228 in → 44 out
e9c50169cfe25469530891a57482484e07c2e40a
JSON Schema Structured Output
(raw) {"hash":"e9c50169cfe25469530891a57482484e07c2e40a"}
liquid/lfm-2-24b-a2b PASS 100% 198 in → 30 out
231e875bc7f9e3b9a57fb7973f0e96b2f81f5e0a
minimax/minimax-m2.5:high PASS 100% 209 in → 130 out (106 reasoning)
81c269338791e900abea9409c53760f1f78f3b66
minimax/minimax-m2.5:high__json_schema PASS 100% 205 in → 223 out (196 reasoning)
fc189001918c1f63febf0fdc814945376b804033
JSON Schema Structured Output
(raw) { "hash": "fc189001918c1f63febf0fdc814945376b804033" }
minimax/minimax-m2.5:low__json_schema PASS 100% 204 in → 168 out (143 reasoning)
347372e761291777159392f1ccc73341541725b9
JSON Schema Structured Output
(raw) { "hash": "347372e761291777159392f1ccc73341541725b9" }
minimax/minimax-m2.5:medium PASS 100% 205 in → 162 out (132 reasoning)
a1142b5f550829ec61249bd4078546791560998d
minimax/minimax-m2.5:medium__json_schema PASS 100% 211 in → 163 out (106 reasoning)
a94f8d2c8773606b56ed8cbf79d488727cf06d2b
JSON Schema Structured Output
(raw) { "hash": "a94f8d2c8773606b56ed8cbf79d488727cf06d2b" }
minimax/minimax-m2.5:xhigh PASS 100% 209 in → 151 out (127 reasoning)
654585eb3bfec70c594e3ee818d1300cf85f1e75
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 205 in → 111 out (77 reasoning)
ec69fe44e223c71ad15a913847ce31349bb18379
JSON Schema Structured Output
(raw) { "hash": "ec69fe44e223c71ad15a913847ce31349bb18379" }
minimax/minimax-m2.7:high PASS 100% 210 in → 262 out (235 reasoning)
fdc107c9736e8c2aef1b64f5bf2c037813495370
minimax/minimax-m2.7:high__json_schema PASS 100% 339 in → 291 out (258 reasoning)
c62a8a19a4744fd423c19d42ad30555a9366c105
JSON Schema Structured Output
(raw) { "hash": "c62a8a19a4744fd423c19d42ad30555a9366c105" }
minimax/minimax-m2.7:low PASS 100% 207 in → 217 out (194 reasoning)
eaf605de1dbcc378d84e1019ca29073ac05bd1cb
minimax/minimax-m2.7:low__json_schema PASS 100% 341 in → 296 out (266 reasoning)
6fd401a64f11b2a36f3946cb2f81501ce079e8ff
JSON Schema Structured Output
(raw) {"hash":"6fd401a64f11b2a36f3946cb2f81501ce079e8ff"}
minimax/minimax-m2.7:medium PASS 100% 209 in → 165 out (129 reasoning)
209bba10239bd5f5b28ab2ddd89af3895292f37d
minimax/minimax-m2.7:medium__json_schema PASS 100% 345 in → 369 out (334 reasoning)
f76f7c9fa8bac9be6a230da01a0d9b2a5c5f8335
JSON Schema Structured Output
(raw) {"hash":"f76f7c9fa8bac9be6a230da01a0d9b2a5c5f8335"}
minimax/minimax-m2.7:xhigh PASS 100% 215 in → 359 out (331 reasoning)
b40c39f30bf8dfa4aee05d7322a23d5be8df3df3
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 338 in → 488 out (457 reasoning)
dea6fbd716939406422aeb36cb118acdb7d3c36e
JSON Schema Structured Output
(raw) { "hash": "dea6fbd716939406422aeb36cb118acdb7d3c36e" }
minimax/minimax-m3:high PASS 100% 350 in → 81 out (44 reasoning)
c37d6de0604e4372e6ee40a06239e5cb150b1d8d
minimax/minimax-m3:high__json_schema PASS 100% 295 in → 31 out (30 reasoning)
d0324b19f8ab08e754d30b007466657a52b7a837
JSON Schema Structured Output
(raw) { "hash": "d0324b19f8ab08e754d30b007466657a52b7a837" }
minimax/minimax-m3:low PASS 100% 345 in → 76 out (52 reasoning)
f218937f1e13cf9378f90723531be8f64e085715
minimax/minimax-m3:low__json_schema PASS 100% 335 in → 32 out (0 reasoning)
1c7b14eb8ea64695d4010a6559f965169c050f61
JSON Schema Structured Output
(raw) { "hash": "1c7b14eb8ea64695d4010a6559f965169c050f61" }
minimax/minimax-m3:medium PASS 100% 349 in → 60 out (44 reasoning)
4b37ba0c88bc74f70392ba240f55c5fc3655f18c
minimax/minimax-m3:medium__json_schema PASS 100% 338 in → 35 out (0 reasoning)
e6df81ae7bcdb75cea89b1c6a96ee6a67cfa7e24
JSON Schema Structured Output
(raw) { "hash": "e6df81ae7bcdb75cea89b1c6a96ee6a67cfa7e24" }
minimax/minimax-m3:xhigh PASS 100% 349 in → 95 out (68 reasoning)
7125faaaffb77e3a75c1dbe9991ec54b28bd4519
minimax/minimax-m3:xhigh__json_schema PASS 100% 338 in → 36 out (0 reasoning)
c4977231c0b6e1ebb835f9e1b35772c1f3be2aa3
JSON Schema Structured Output
(raw) { "hash": "c4977231c0b6e1ebb835f9e1b35772c1f3be2aa3" }
mistralai/devstral-2512 PASS 100% 206 in → 38 out
da7c00ef27f9d99d0a32b6c397b2a03f08d78bf2
mistralai/devstral-2512__json_schema PASS 100% 205 in → 41 out
743bad866b3e8c449b3533fd3185cee70d0fea66
JSON Schema Structured Output
(raw) {"hash": "743bad866b3e8c449b3533fd3185cee70d0fea66"}
mistralai/mistral-medium-3-5:high PASS 100% 214 in → 283 out (208 reasoning)
89f2472be2f1ff3484aee023fe11ff7ee5a57186
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 205 in → 249 out (158 reasoning)
bc7069913c6c198a132b409065f4a3e73f79feb7
JSON Schema Structured Output
(raw) {"hash": "bc7069913c6c198a132b409065f4a3e73f79feb7"}
mistralai/mistral-medium-3-5:low PASS 100% 214 in → 241 out (164 reasoning)
3aa391e88743fbf343cfa8cb2a0419be14cc187e
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 198 in → 323 out (224 reasoning)
cd6c865cddaa5dba1ea4cbb3c9bcec7ffc93f4ec
JSON Schema Structured Output
(raw) {"hash": "cd6c865cddaa5dba1ea4cbb3c9bcec7ffc93f4ec"}
mistralai/mistral-medium-3-5:medium PASS 100% 217 in → 286 out (206 reasoning)
98c4e4b75e0e1797f0980a5c04cca73b3fb97721
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 201 in → 766 out (678 reasoning)
eeabe51f969eb9d074d530bd89fb96916b95b59b
JSON Schema Structured Output
(raw) {"hash": "eeabe51f969eb9d074d530bd89fb96916b95b59b"}
mistralai/mistral-medium-3-5:none PASS 100% 215 in → 37 out (0 reasoning)
ebce903c029e2e945ce74403f20c55b5adf2c5d1
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 205 in → 44 out (0 reasoning)
eb3595645a2f3b8f020970e605514914eb93e0d4
JSON Schema Structured Output
(raw) {"hash": "eb3595645a2f3b8f020970e605514914eb93e0d4"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 213 in → 264 out (201 reasoning)
1dd2fb76b8e806c7fd3e1ceff53ddc0378959fd7
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 206 in → 352 out (218 reasoning)
75e2709c30a51c633a55b78bd7b800a0710eb0f7
JSON Schema Structured Output
(raw) {"hash": "75e2709c30a51c633a55b78bd7b800a0710eb0f7"}
moonshotai/kimi-k2.6:high PASS 100% 181 in → 182 out (156 reasoning)
4b2d7045931e4cb505bb3d9790c7048e844557d8
moonshotai/kimi-k2.6:high__json_schema PASS 100% 183 in → 201 out (162 reasoning)
b2e82e49f63d31366569a9a56c248e32f3dd6b4c
JSON Schema Structured Output
(raw) { "hash": "b2e82e49f63d31366569a9a56c248e32f3dd6b4c" }
moonshotai/kimi-k2.6:low PASS 100% 185 in → 248 out (162 reasoning)
b13a3e29faa3c6d3c8f67d7411c1a62f567f94a0
moonshotai/kimi-k2.6:low__json_schema PASS 100% 178 in → 574 out (202 reasoning)
630d802e0ed5f154b09638d0a4f886dcf410921d
JSON Schema Structured Output
(raw) { "hash": "630d802e0ed5f154b09638d0a4f886dcf410921d" }
moonshotai/kimi-k2.6:medium PASS 100% 179 in → 242 out (217 reasoning)
d08d3bef9770787da099f90c2dbeaa2741ed6c3e
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 183 in → 185 out (154 reasoning)
73e0b6873c5b7458955d5d6afe02a2e9bcb4eef1
JSON Schema Structured Output
(raw) {"hash":"73e0b6873c5b7458955d5d6afe02a2e9bcb4eef1"}
moonshotai/kimi-k2.6:xhigh PASS 100% 180 in → 235 out (210 reasoning)
8968c5b39e6fe2439c09b269a59f370cd065f647
moonshotai/kimi-k2.7-code:high PASS 100% 175 in → 51 out (21 reasoning)
de251a42fa22edc8fb25545c423c669000aa416c
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 180 in → 176 out (128 reasoning)
fe1d1057b0a291dfadceeb0bba3c50fc1b92bb70
JSON Schema Structured Output
(raw) { "hash": "fe1d1057b0a291dfadceeb0bba3c50fc1b92bb70" }
moonshotai/kimi-k2.7-code:low PASS 100% 182 in → 76 out (36 reasoning)
58b72b0135ccb9002a6cbe9e1028782c2d4adf1d
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 262 in → 164 out (133 reasoning)
2375d2c67db0cf954e33055468ed8f57d8b99e57
JSON Schema Structured Output
(raw) {"hash": "2375d2c67db0cf954e33055468ed8f57d8b99e57"}
moonshotai/kimi-k2.7-code:medium PASS 100% 182 in → 83 out (56 reasoning)
1b65ea5b4aa6b65f8a9f9de014dfdbeda6585ccd
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 177 in → 72 out (43 reasoning)
08d199360ba7648eaf5186d8507940b883fbbb9e
JSON Schema Structured Output
(raw) { "hash": "08d199360ba7648eaf5186d8507940b883fbbb9e" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 180 in → 106 out (81 reasoning)
4e561cc1384a0b6bf21aa4c11092464fac1a5808
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 176 in → 147 out (111 reasoning)
85c34129974adb588ae36a7023838804a01bdf5b
JSON Schema Structured Output
(raw) { "hash": "85c34129974adb588ae36a7023838804a01bdf5b" }
moonshotai/kimi-k3:high PASS 100% 256 in → 165 out (127 reasoning)
a12d3ff25022d135665ce9b7e11fb3b92766359c
moonshotai/kimi-k3:high__json_schema PASS 100% 259 in → 105 out (65 reasoning)
60f4f9d41179e28abad19a38d4ad97492f4d03d6
moonshotai/kimi-k3:low PASS 100% 255 in → 134 out (98 reasoning)
23b315dcf151d81cc7b32f2071aac44aa0149c87
moonshotai/kimi-k3:low__json_schema PASS 100% 254 in → 104 out (69 reasoning)
ffd0d4ed5877fdd997a7131adbeb6bdbfc60185d
moonshotai/kimi-k3:max PASS 100% 256 in → 163 out (126 reasoning)
78bd6686100bde9d4354ff6db656bf0fb1a518f2
moonshotai/kimi-k3:max__json_schema PASS 100% 257 in → 112 out (74 reasoning)
44486e4bb22903b663b3a666fc3d12ea68c5bcb9
moonshotai/kimi-k3:medium PASS 100% 256 in → 111 out (73 reasoning)
f1768475ec94a5ea2baf67690d7be6d59b617527
moonshotai/kimi-k3:medium__json_schema PASS 100% 255 in → 104 out (68 reasoning)
7f933862f42d5fff517d4b35256eeefbe98ac997
moonshotai/kimi-k3:xhigh PASS 100% 255 in → 106 out (70 reasoning)
835a965c355e78e9cda329b5285060a2dbf5b118
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 257 in → 112 out (74 reasoning)
8e7221f7f1cb2f59773222880698a84d6713538e
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 217 in → 356 out (221 reasoning)
c818c017868ee3941532fcd07a9986ad20f8fff1
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 217 in → 471 out (206 reasoning)
edbdc8d76be814251054df04eb9d72c53a796503
JSON Schema Structured Output
(raw) { "hash": "edbdc8d76be814251054df04eb9d72c53a796503" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 220 in → 244 out (118 reasoning)
1186882a867684fbae4287e7e20c339de6313ba2
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 216 in → 183 out (101 reasoning)
b7f070febb55cdf981354d93bffe5a8ce743052c
JSON Schema Structured Output
(raw) { "hash": "b7f070febb55cdf981354d93bffe5a8ce743052c" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 217 in → 130 out (78 reasoning)
6a06f830a8e5c1defc9698177957ecc5880abebe
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 221 in → 623 out (236 reasoning)
e0a9b81513e5f0df75d8c5a0a6939d1d96751007
JSON Schema Structured Output
(raw) { "hash": "e0a9b81513e5f0df75d8c5a0a6939d1d96751007" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 218 in → 36 out (0 reasoning)
7a884ec4cab6b2707db59bb476695ee7b849b702
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 216 in → 42 out (0 reasoning)
83f5423deadce4237dd94dbf36f21388e76614ac
JSON Schema Structured Output
(raw) { "hash": "83f5423deadce4237dd94dbf36f21388e76614ac" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 220 in → 219 out (130 reasoning)
4d7432b907fb94a4541f0bcd569dd9c109405e6a
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 218 in → 311 out (192 reasoning)
037758ff005539e318cf49ce4b05c44bfc0b4ed9
JSON Schema Structured Output
(raw) { "hash": "037758ff005539e318cf49ce4b05c44bfc0b4ed9" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 219 in → 114 out (59 reasoning)
278628ed924d3e0ad955fca1f972246da19d5295
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 218 in → 170 out (88 reasoning)
569ea005e025109b124353803638e1f1d20fa2ca
JSON Schema Structured Output
(raw) { "hash": "569ea005e025109b124353803638e1f1d20fa2ca" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 219 in → 157 out (71 reasoning)
5ae76808bf1a27f146e84142be5e55d9e1ce5113
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 219 in → 109 out (50 reasoning)
c5a82c6116ba5c180e07779e069ba073372f2df4
JSON Schema Structured Output
(raw) {"hash": "c5a82c6116ba5c180e07779e069ba073372f2df4"}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 217 in → 109 out (52 reasoning)
0abcb08fd4a780bbef151f960521cf5e6d87c00c
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 220 in → 254 out (177 reasoning)
1e75b67be0362ac93087cfd8b60c0a2704457520
JSON Schema Structured Output
(raw) { "hash": "1e75b67be0362ac93087cfd8b60c0a2704457520" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 222 in → 40 out (0 reasoning)
511466a6429d7b6372879c857bf2157c8c8b996c
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 220 in → 46 out (0 reasoning)
94290089b27cbba40dd37f66436365180e8b6977
JSON Schema Structured Output
(raw) { "hash": "94290089b27cbba40dd37f66436365180e8b6977" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 216 in → 703 out (558 reasoning)
a1c02185ed7cb90d2ab5a77a361916767141ae90
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 216 in → 300 out (213 reasoning)
1195f54eb1d9c6acfb007b1e49275bcf799ec206
JSON Schema Structured Output
(raw) { "hash": "1195f54eb1d9c6acfb007b1e49275bcf799ec206" }
openai/gpt-5.4-mini:high PASS 100% 181 in → 92 out (60 reasoning)
2ae6a75da9f4895374d9a93f5f52465e34d6ffc9
openai/gpt-5.4-mini:high__json_schema PASS 100% 210 in → 134 out (98 reasoning)
9e7065ca61c56160dccc902695d8d32c8940addc
JSON Schema Structured Output
(raw) {"hash":"9e7065ca61c56160dccc902695d8d32c8940addc"}
openai/gpt-5.4-mini:low PASS 100% 175 in → 36 out (9 reasoning)
b2589e673acc084b419cdd7d1bef113d20a77695
openai/gpt-5.4-mini:low__json_schema PASS 100% 210 in → 46 out (9 reasoning)
a77eeebeb5ecb699746661aab2680c3fd65a6ef6
JSON Schema Structured Output
(raw) {"hash":"a77eeebeb5ecb699746661aab2680c3fd65a6ef6"}
openai/gpt-5.4-mini:medium PASS 100% 173 in → 34 out (9 reasoning)
e82786ae781bdbec721b21a83910e583ed692905
openai/gpt-5.4-mini:medium__json_schema PASS 100% 214 in → 72 out (32 reasoning)
11ac02c6a063c8c33cd1746bd9f2fc2b5df2a663
JSON Schema Structured Output
(raw) {"hash":"11ac02c6a063c8c33cd1746bd9f2fc2b5df2a663"}
openai/gpt-5.4-mini:none PASS 100% 177 in → 26 out (0 reasoning)
51f364986126cd3044c1eb9891cfaace8e5962e0
openai/gpt-5.4-mini:none__json_schema PASS 100% 215 in → 39 out (0 reasoning)
6d7a47f6a0c7f0c5ec7ace920a0dfacdb739de17
JSON Schema Structured Output
(raw) {"hash":"6d7a47f6a0c7f0c5ec7ace920a0dfacdb739de17"}
openai/gpt-5.4-mini:xhigh PASS 100% 176 in → 138 out (111 reasoning)
80b83c60eefbe56002406c5f772f7a154efbd940
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 212 in → 3,615 out (3,577 reasoning)
149d52d157b17f9eb1a11b66b71f37a65ec443ef
JSON Schema Structured Output
(raw) {"hash":"149d52d157b17f9eb1a11b66b71f37a65ec443ef"}
openai/gpt-5.4-nano:high PASS 100% 180 in → 79 out (48 reasoning)
6d3c367529722bf6c845f34683d9f267d01c9c8f
openai/gpt-5.4-nano:high__json_schema PASS 100% 209 in → 88 out (53 reasoning)
49acf95b6744ba3708b11103af1578d5827a3785
JSON Schema Structured Output
(raw) {"hash":"49acf95b6744ba3708b11103af1578d5827a3785"}
openai/gpt-5.4-nano:low PASS 100% 178 in → 27 out (0 reasoning)
5f695de11f575e11cd6be2d01c7dac70ffc938d5
openai/gpt-5.4-nano:low__json_schema PASS 100% 211 in → 35 out (0 reasoning)
0db04d40c498f93f01d50ab20770217e04cf98b0
JSON Schema Structured Output
(raw) {"hash":"0db04d40c498f93f01d50ab20770217e04cf98b0"}
openai/gpt-5.4-nano:medium PASS 100% 179 in → 68 out (38 reasoning)
169ca05ff27b82f2d4d8b6bfeac12ae65187c345
openai/gpt-5.4-nano:medium__json_schema PASS 100% 213 in → 75 out (36 reasoning)
4859407a22a62a4427ecb432f1d0d8110f3805b7
JSON Schema Structured Output
(raw) {"hash":"4859407a22a62a4427ecb432f1d0d8110f3805b7"}
openai/gpt-5.4-nano:none PASS 100% 179 in → 28 out (0 reasoning)
75708fe8cb2c096928e7ae58a5ea3b44a9afc627
openai/gpt-5.4-nano:none__json_schema PASS 100% 210 in → 34 out (0 reasoning)
07318eeb87c40b07e3b5ba520a423109cbfb2816
JSON Schema Structured Output
(raw) {"hash":"07318eeb87c40b07e3b5ba520a423109cbfb2816"}
openai/gpt-5.4-nano:xhigh PASS 100% 179 in → 152 out (122 reasoning)
09b3dad823a4b2029b9f0216de4bd72394e62c31
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 208 in → 616 out (582 reasoning)
dfa80860129849cdd9cb4b07c6149915f621ec66
JSON Schema Structured Output
(raw) {"hash":"dfa80860129849cdd9cb4b07c6149915f621ec66"}
openai/gpt-5.4:high PASS 100% 173 in → 153 out (128 reasoning)
d625f458898f6da387dbccb163dab83814be68b9
openai/gpt-5.4:high__json_schema PASS 100% 208 in → 107 out (73 reasoning)
bea73dd5ca6ec4ae06c811237a354ae744ef4910
JSON Schema Structured Output
(raw) {"hash":"bea73dd5ca6ec4ae06c811237a354ae744ef4910"}
openai/gpt-5.4:low PASS 100% 180 in → 41 out (9 reasoning)
c376cca6c46e8f5f432b8a1c606bd58e260b33b4
openai/gpt-5.4:low__json_schema PASS 100% 208 in → 44 out (9 reasoning)
e32edd6011bd7ed7b2716da5088a28383447e205
JSON Schema Structured Output
(raw) {"hash":"e32edd6011bd7ed7b2716da5088a28383447e205"}
openai/gpt-5.4:medium PASS 100% 176 in → 76 out (48 reasoning)
afe86c228b430eafc61e98a6bdd21453ad2b1933
openai/gpt-5.4:medium__json_schema PASS 100% 211 in → 69 out (32 reasoning)
72248d018ec619de57b66314ac0df8a6fad9a5c7
JSON Schema Structured Output
(raw) {"hash":"72248d018ec619de57b66314ac0df8a6fad9a5c7"}
openai/gpt-5.4:none PASS 100% 178 in → 28 out (0 reasoning)
f8e98e34cbd10e32291cd1570420edac7e7539e8
openai/gpt-5.4:none__json_schema PASS 100% 210 in → 34 out (0 reasoning)
2b1f61c2cee2cd4cea078ea516210326839eec2f
JSON Schema Structured Output
(raw) {"hash":"2b1f61c2cee2cd4cea078ea516210326839eec2f"}
openai/gpt-5.4:xhigh PASS 100% 178 in → 97 out (67 reasoning)
f06d88f9e727605429b970e31d19fcd6d600d4fc
openai/gpt-5.4:xhigh__json_schema PASS 100% 209 in → 97 out (62 reasoning)
2831338005e42e419a97afc97abae3141b39abec
JSON Schema Structured Output
(raw) {"hash":"2831338005e42e419a97afc97abae3141b39abec"}
openai/gpt-5.5:high PASS 100% 176 in → 40 out (13 reasoning)
08917656b864512d8adc38678a2f178dac7e9b84
openai/gpt-5.5:high__json_schema PASS 100% 216 in → 40 out (0 reasoning)
6ee1ccf18449dc6f0c8b0d554cc8cbe1a5e55a35
JSON Schema Structured Output
(raw) {"hash":"6ee1ccf18449dc6f0c8b0d554cc8cbe1a5e55a35"}
openai/gpt-5.5:low PASS 100% 175 in → 24 out (0 reasoning)
2716fff396bd2c6037643ecda9557d4c451cf288
openai/gpt-5.5:low__json_schema PASS 100% 212 in → 36 out (0 reasoning)
092f412021b8c51e9aa46f028e86e966b7b4f045
JSON Schema Structured Output
(raw) {"hash":"092f412021b8c51e9aa46f028e86e966b7b4f045"}
openai/gpt-5.5:medium PASS 100% 175 in → 36 out (10 reasoning)
3f3705ca0d0965182590117fff28d844254e765a
openai/gpt-5.5:medium__json_schema PASS 100% 211 in → 35 out (0 reasoning)
7fda524f7310716c136df90ec09d7ef4c7de131f
JSON Schema Structured Output
(raw) {"hash":"7fda524f7310716c136df90ec09d7ef4c7de131f"}
openai/gpt-5.5:none PASS 100% 176 in → 26 out (0 reasoning)
b7ae54653d3aee2c00946532d515011563b09e65
openai/gpt-5.5:none__json_schema PASS 100% 208 in → 32 out (0 reasoning)
78570feca782ab4202d8152a4395fad0cdbed794
JSON Schema Structured Output
(raw) {"hash":"78570feca782ab4202d8152a4395fad0cdbed794"}
openai/gpt-5.5:xhigh PASS 100% 174 in → 77 out (51 reasoning)
c92f50295632edcccbceae489b32276a22a640eb
openai/gpt-5.5:xhigh__json_schema PASS 100% 205 in → 62 out (30 reasoning)
cc76542e10443e86ec58def503bb687b837aa960
JSON Schema Structured Output
(raw) {"hash":"cc76542e10443e86ec58def503bb687b837aa960"}
openai/gpt-5.6-luna:high PASS 100% 177 in → 26 out (0 reasoning)
8458a21c578a66f204bc43b8ef430a8cf069ad62
openai/gpt-5.6-luna:high__json_schema PASS 100% 212 in → 36 out (0 reasoning)
73640493b74fa2a3e4909f800e2c887d62b2695f
JSON Schema Structured Output
(raw) {"hash":"73640493b74fa2a3e4909f800e2c887d62b2695f"}
openai/gpt-5.6-luna:low PASS 100% 177 in → 27 out (0 reasoning)
bc2e9c20bf66af5b17028fc6da97fe7074746ce1
openai/gpt-5.6-luna:low__json_schema PASS 100% 212 in → 36 out (0 reasoning)
6d9d0fc0ad146736e4ebb0e29a67333f1f613265
JSON Schema Structured Output
(raw) {"hash":"6d9d0fc0ad146736e4ebb0e29a67333f1f613265"}
openai/gpt-5.6-luna:max PASS 100% 177 in → 61 out (32 reasoning)
bc0e021db6d1cab31b23dd72dc0a60ddfeb0c788
openai/gpt-5.6-luna:max__json_schema PASS 100% 213 in → 38 out (0 reasoning)
d6f4c2dae6f7d64a11b07ca00ef0796053042b5b
JSON Schema Structured Output
(raw) {"hash":"d6f4c2dae6f7d64a11b07ca00ef0796053042b5b"}
openai/gpt-5.6-luna:medium PASS 100% 177 in → 26 out (0 reasoning)
3242997a32dfc464b0b4e90a37c0892edd089102
openai/gpt-5.6-luna:medium__json_schema PASS 100% 211 in → 35 out (0 reasoning)
1f74cf9053db60bf13baccf78e56950b777b7ae6
JSON Schema Structured Output
(raw) {"hash":"1f74cf9053db60bf13baccf78e56950b777b7ae6"}
openai/gpt-5.6-luna:none PASS 100% 176 in → 25 out (0 reasoning)
6f91d7f51b32a4838098560cc35431887850c814
openai/gpt-5.6-luna:none__json_schema PASS 100% 217 in → 41 out (0 reasoning)
3f2f16562c901e8f6349f8fb9f3c5c06cfa4dd5f
JSON Schema Structured Output
(raw) {"hash":"3f2f16562c901e8f6349f8fb9f3c5c06cfa4dd5f"}
openai/gpt-5.6-luna:xhigh PASS 100% 177 in → 48 out (20 reasoning)
61035fdd07ca0cf5fca68c27815bc6edae72268c
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 213 in → 37 out (0 reasoning)
169d42c8e030475e076a31a4cab9f6be3b6f9569
JSON Schema Structured Output
(raw) {"hash":"169d42c8e030475e076a31a4cab9f6be3b6f9569"}
openai/gpt-5.6-sol:high PASS 100% 178 in → 28 out (0 reasoning)
ab0904a48b8048f539a4c1a901556f6bb522a5bf
openai/gpt-5.6-sol:high__json_schema PASS 100% 209 in → 33 out (0 reasoning)
21240025de085ff9f60cfde31fdba75fba5298a2
JSON Schema Structured Output
(raw) {"hash":"21240025de085ff9f60cfde31fdba75fba5298a2"}
openai/gpt-5.6-sol:low PASS 100% 175 in → 24 out (0 reasoning)
26ad489ab442011c0ae929a9413736ee71f1e589
openai/gpt-5.6-sol:low__json_schema PASS 100% 215 in → 39 out (0 reasoning)
4353e25e52a926a188b9b8263e0b9545f67b0f8c
JSON Schema Structured Output
(raw) {"hash":"4353e25e52a926a188b9b8263e0b9545f67b0f8c"}
openai/gpt-5.6-sol:max PASS 100% 175 in → 24 out (0 reasoning)
1cf7a806442177317b4f18045946220f9612753f
openai/gpt-5.6-sol:max__json_schema PASS 100% 210 in → 34 out (0 reasoning)
1a4767608fe36f986ea904b9d137c35c256abaf8
JSON Schema Structured Output
(raw) {"hash":"1a4767608fe36f986ea904b9d137c35c256abaf8"}
openai/gpt-5.6-sol:medium PASS 100% 179 in → 28 out (0 reasoning)
3ecb98c917cd2902c620abb542cf7e53b0e5fb4c
openai/gpt-5.6-sol:medium__json_schema PASS 100% 211 in → 36 out (0 reasoning)
d2722416919a74aa85f0a5ed55ccea8177d1ce7e
JSON Schema Structured Output
(raw) {"hash":"d2722416919a74aa85f0a5ed55ccea8177d1ce7e"}
openai/gpt-5.6-sol:none PASS 100% 175 in → 24 out (0 reasoning)
9973533b715a1886a17782485bcaa2836c49344e
openai/gpt-5.6-sol:none__json_schema PASS 100% 212 in → 37 out (0 reasoning)
b6dbb7a9662d6b2c836ad3cd93917f500c54304f
JSON Schema Structured Output
(raw) {"hash":"b6dbb7a9662d6b2c836ad3cd93917f500c54304f"}
openai/gpt-5.6-sol:xhigh PASS 100% 178 in → 28 out (0 reasoning)
f97c79ae8013d626380f13a7cd22a71f6076d6ca
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 212 in → 36 out (0 reasoning)
4f2e050f418870d067c4c6c37edda95a462d475c
JSON Schema Structured Output
(raw) {"hash":"4f2e050f418870d067c4c6c37edda95a462d475c"}
openai/gpt-5.6-terra:high PASS 100% 178 in → 28 out (0 reasoning)
cb882f7b0c91aee87787db962c7355b480dcb3b9
openai/gpt-5.6-terra:high__json_schema PASS 100% 213 in → 37 out (0 reasoning)
0e71bb9de696fd1b9d70a02f91639b10605c66f6
JSON Schema Structured Output
(raw) {"hash":"0e71bb9de696fd1b9d70a02f91639b10605c66f6"}
openai/gpt-5.6-terra:low PASS 100% 178 in → 28 out (0 reasoning)
f376a1bb14a38524c7ad6dfb4c3ba0f225332496
openai/gpt-5.6-terra:low__json_schema PASS 100% 206 in → 30 out (0 reasoning)
99f8ca050005cc830ca72ffc3fe582db51771ebb
JSON Schema Structured Output
(raw) {"hash":"99f8ca050005cc830ca72ffc3fe582db51771ebb"}
openai/gpt-5.6-terra:max PASS 100% 177 in → 26 out (0 reasoning)
57735a09d12fe201248d1770021f5d0cbaba2c63
openai/gpt-5.6-terra:max__json_schema PASS 100% 210 in → 34 out (0 reasoning)
64768c777e0e288396e78091b11bba8059b57b35
JSON Schema Structured Output
(raw) {"hash":"64768c777e0e288396e78091b11bba8059b57b35"}
openai/gpt-5.6-terra:medium PASS 100% 182 in → 32 out (0 reasoning)
ca7e84d9f7b8599e8814b54f7930bbcf7e8d66c0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 211 in → 35 out (0 reasoning)
462d08f4297c8e57de69668560db76aaa0a76a9f
JSON Schema Structured Output
(raw) {"hash":"462d08f4297c8e57de69668560db76aaa0a76a9f"}
openai/gpt-5.6-terra:none PASS 100% 177 in → 26 out (0 reasoning)
7658630119ac9a54708a1cf02b2338ba44c78a85
openai/gpt-5.6-terra:none__json_schema PASS 100% 215 in → 39 out (0 reasoning)
8ea9e7b47292ecc0d3e68f6d2451d2ab29bf7dbe
JSON Schema Structured Output
(raw) {"hash":"8ea9e7b47292ecc0d3e68f6d2451d2ab29bf7dbe"}
openai/gpt-5.6-terra:xhigh PASS 100% 182 in → 32 out (0 reasoning)
a48a6c3f55eb7e4b9007ba1bdec0c4cffd5fa45d
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 215 in → 39 out (0 reasoning)
9312759eb37dce8d71223e9a5f1c2bca8d1ea82a
JSON Schema Structured Output
(raw) {"hash":"9312759eb37dce8d71223e9a5f1c2bca8d1ea82a"}
openai/gpt-oss-120b:high PASS 100% 225 in → 174 out (115 reasoning)
794a45085f1c52abc8306780578a48e2dc91b95a
openai/gpt-oss-120b:high__json_schema PASS 100% 238 in → 429 out (314 reasoning)
0eef022448680290bb933b6ecb7551969e0647c8
JSON Schema Structured Output
(raw) { "hash": "0eef022448680290bb933b6ecb7551969e0647c8" }
openai/gpt-oss-120b:low PASS 100% 224 in → 61 out (18 reasoning)
24ba1d2c926a40417e7448710a49610d658412ef
openai/gpt-oss-120b:low__json_schema PASS 100% 237 in → 90 out (39 reasoning)
b6cdac8f70d0ba437f5ff359dd8faa55d090a3ad
JSON Schema Structured Output
(raw) { "hash": "b6cdac8f70d0ba437f5ff359dd8faa55d090a3ad" }
openai/gpt-oss-120b:medium PASS 100% 239 in → 157 out (125 reasoning)
ad15ac3851beb1e75a6204f6fb1093f86478f0f0
openai/gpt-oss-120b:medium__json_schema PASS 100% 223 in → 118 out (75 reasoning)
41b3ad741e268ba140af03800fe13f16666294a8
JSON Schema Structured Output
(raw) { "hash": "41b3ad741e268ba140af03800fe13f16666294a8" }
openai/gpt-oss-120b:xhigh PASS 100% 244 in → 169 out (141 reasoning)
3c166e1668a4f00e5a8f0843a8b6c82182866d0c
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 220 in → 427 out (356 reasoning)
70952c17575018804140e034a66ffc0608119729
JSON Schema Structured Output
(raw) { "hash": "70952c17575018804140e034a66ffc0608119729" }
openai/gpt-oss-20b:high PASS 100% 241 in → 268 out (179 reasoning)
0f37f93b86aa07a93c69d9c826b505a4a928152f
openai/gpt-oss-20b:low PASS 100% 222 in → 66 out (26 reasoning)
d10697c5f9f630fd162ad778d579038956f47137
openai/gpt-oss-20b:low__json_schema PASS 100% 285 in → 100 out (60 reasoning)
225bcf8ed56899287982b045e12a43a722a65f12
JSON Schema Structured Output
(raw) {"hash":"225bcf8ed56899287982b045e12a43a722a65f12"}
openai/gpt-oss-20b:medium PASS 100% 225 in → 128 out (75 reasoning)
08caa50887a6696030c9a948a64bdab9425bc8fa
openai/gpt-oss-20b:medium__json_schema PASS 100% 240 in → 187 out (146 reasoning)
50a2eaee83c5eba371619e4e4998a709a6e500b8
JSON Schema Structured Output
(raw) { "hash": "50a2eaee83c5eba371619e4e4998a709a6e500b8" }
openai/gpt-oss-20b:xhigh PASS 100% 235 in → 437 out (283 reasoning)
fad194ef79503aaf6241488278a4dc4c0298942e
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 227 in → 366 out (235 reasoning)
74497a2766e0a320225d4a04995f4384d4d5d872
JSON Schema Structured Output
(raw) {"hash":"74497a2766e0a320225d4a04995f4384d4d5d872"}
poolside/laguna-m.1:high PASS 100% 214 in → 279 out (241 reasoning)
0e1e91770417c58dafd105f089bc3a24d37fef3b
poolside/laguna-m.1:high__json_schema PASS 100% 212 in → 233 out (188 reasoning)
ee42222352e03e35dab65906dc0dfc30a088f157
JSON Schema Structured Output
(raw) { "hash": "ee42222352e03e35dab65906dc0dfc30a088f157" }
poolside/laguna-m.1:low PASS 100% 218 in → 1,752 out (1,711 reasoning)
1334147f984f57b89035a0926cc7127786850582
poolside/laguna-m.1:low__json_schema PASS 100% 216 in → 342 out (297 reasoning)
33d04f0b94d623fc71648d305832ef7b6de0928e
JSON Schema Structured Output
(raw) {"hash": "33d04f0b94d623fc71648d305832ef7b6de0928e"}
poolside/laguna-m.1:medium PASS 100% 211 in → 432 out (396 reasoning)
ab70e06c9ff697ddf8614fe0ee39ec288f881d86
poolside/laguna-m.1:medium__json_schema PASS 100% 216 in → 231 out (181 reasoning)
aee9f108a8ea92a64289c41389278666b5d96a2e
JSON Schema Structured Output
(raw) { "hash": "aee9f108a8ea92a64289c41389278666b5d96a2e" }
poolside/laguna-m.1:none PASS 100% 215 in → 39 out (0 reasoning)
c1b94b40e9e7cc823bb43018f02698b899ca5e77
poolside/laguna-m.1:none__json_schema PASS 100% 217 in → 44 out (0 reasoning)
39090600047860dec42080621613922b787d1c46
JSON Schema Structured Output
(raw) {"hash": "39090600047860dec42080621613922b787d1c46"}
poolside/laguna-m.1:xhigh PASS 100% 213 in → 925 out (887 reasoning)
f7ab77a0607e1d678a3c418de74b7eb5aa3931fe
poolside/laguna-m.1:xhigh__json_schema PASS 100% 213 in → 274 out (229 reasoning)
4d5327da3c4773a3f173c16ceeac5f3d0a64cfd5
JSON Schema Structured Output
(raw) { "hash": "4d5327da3c4773a3f173c16ceeac5f3d0a64cfd5" }
poolside/laguna-xs-2.1:high PASS 100% 218 in → 309 out (270 reasoning)
1001302016b45c8b24e2a0777381fdefd0a8d4d5
poolside/laguna-xs-2.1:high__json_schema PASS 100% 218 in → 653 out (603 reasoning)
36a23752331fa1fe4461498d648a53952b2b4765
JSON Schema Structured Output
(raw) { "hash": "36a23752331fa1fe4461498d648a53952b2b4765" }
poolside/laguna-xs-2.1:low PASS 100% 214 in → 213 out (178 reasoning)
dfc36f238833f15ef61b4ef0c2ad9594864080ee
poolside/laguna-xs-2.1:low__json_schema PASS 100% 209 in → 273 out (231 reasoning)
af6311dee30c365bbb5386f6bbd64ac44cb3dc16
JSON Schema Structured Output
(raw) { "hash": "af6311dee30c365bbb5386f6bbd64ac44cb3dc16" }
poolside/laguna-xs-2.1:medium PASS 100% 217 in → 535 out (498 reasoning)
072c90726cb5f2156f8754321a54557852da69bf
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 217 in → 507 out (458 reasoning)
047b2cea8f141289b76b9e9818399881840ad1c1
JSON Schema Structured Output
(raw) { "hash": "047b2cea8f141289b76b9e9818399881840ad1c1" }
poolside/laguna-xs-2.1:none PASS 100% 215 in → 72 out (0 reasoning)
32753051a3473dd2597f0c25baeedf488d74fc74 </think>32753051a3473dd2597f0c25baeedf488d74fc74
poolside/laguna-xs-2.1:none__json_schema PASS 100% 218 in → 49 out (0 reasoning)
d553426898b1716f535c7cd657b9943d8f890e6b
JSON Schema Structured Output
(raw) { "hash": "d553426898b1716f535c7cd657b9943d8f890e6b" }
poolside/laguna-xs-2.1:xhigh PASS 100% 215 in → 281 out (244 reasoning)
c90e574d1186e527901cf2afc64f43459c0b0ea5
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 214 in → 304 out (258 reasoning)
68ad66aac007c43c74716ba4cf901f8ac32b033a
JSON Schema Structured Output
(raw) { "hash": "68ad66aac007c43c74716ba4cf901f8ac32b033a" }
poolside/laguna-xs.2:high PASS 100% 250 in → 150 out (113 reasoning)
ca3072ae906b92c891eb73214650fab100ef111d
poolside/laguna-xs.2:high__json_schema PASS 100% 253 in → 223 out (181 reasoning)
6e8a87450d91220ad74095b1c0fdc561f70ea040
JSON Schema Structured Output
(raw) {"hash":"6e8a87450d91220ad74095b1c0fdc561f70ea040"}
poolside/laguna-xs.2:low PASS 100% 252 in → 195 out (157 reasoning)
3a1b2d88fe08d5b8068492ab800eabaf8472a40d
poolside/laguna-xs.2:low__json_schema PASS 100% 243 in → 182 out (144 reasoning)
dcd5fed231059cb81d8fb1789cf90caffdacdda6
JSON Schema Structured Output
(raw) { "hash": "dcd5fed231059cb81d8fb1789cf90caffdacdda6" }
poolside/laguna-xs.2:medium PASS 100% 257 in → 213 out (170 reasoning)
93864425c0a3012d411117e2689b354e21b27a76
poolside/laguna-xs.2:medium__json_schema PASS 100% 250 in → 244 out (204 reasoning)
c34bcf9c08f86de2cc09608d11fe3f14a20ae3c8
JSON Schema Structured Output
(raw) {"hash":"c34bcf9c08f86de2cc09608d11fe3f14a20ae3c8"}
poolside/laguna-xs.2:none PASS 100% 252 in → 37 out (0 reasoning)
9255440ee9a3ddaee7005851490430ce450d6c74
poolside/laguna-xs.2:none__json_schema PASS 100% 253 in → 43 out (0 reasoning)
f773b7f115c4563d2d2e610bd54a7fb98b5c65aa
JSON Schema Structured Output
(raw) {"hash": "f773b7f115c4563d2d2e610bd54a7fb98b5c65aa"}
poolside/laguna-xs.2:xhigh PASS 100% 243 in → 180 out (151 reasoning)
9ffa5849fce32e7dbbaa8cfdeaac751fd8df59da
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 249 in → 219 out (181 reasoning)
3b04e3cc4bb657ac55f8a20fd7d3658fdddeba10
JSON Schema Structured Output
(raw) {"hash":"3b04e3cc4bb657ac55f8a20fd7d3658fdddeba10"}
qwen/qwen3.6-27b:high PASS 100% 208 in → 223 out (1 reasoning)
7dfcf0ee1db53720c7249ba49b4a81efba4ac273
qwen/qwen3.6-27b:high__json_schema PASS 100% 213 in → 856 out (556 reasoning)
02ebdf9a411e61b8e28c4b7a9196d6ee84e573b7
JSON Schema Structured Output
(raw) { "hash": "02ebdf9a411e61b8e28c4b7a9196d6ee84e573b7" }
qwen/qwen3.6-27b:low PASS 100% 210 in → 1,055 out (736 reasoning)
7ec4ea3e2ebc586151245e8f89474deec8b739fe
qwen/qwen3.6-27b:low__json_schema PASS 100% 211 in → 236 out (120 reasoning)
3a9e39bc8e73db35996b3dd201ce05e9be24b949
JSON Schema Structured Output
(raw) { "hash": "3a9e39bc8e73db35996b3dd201ce05e9be24b949" }
qwen/qwen3.6-27b:medium PASS 100% 212 in → 306 out (179 reasoning)
7d05ad85644f7f8d95a20b1b1992538acdae0c3e
qwen/qwen3.6-27b:medium__json_schema PASS 100% 214 in → 1,173 out (789 reasoning)
8b0fdf23817b3999221572037a0431492f674a2b
JSON Schema Structured Output
(raw) { "hash": "8b0fdf23817b3999221572037a0431492f674a2b" }
qwen/qwen3.6-27b:none PASS 100% 212 in → 35 out (0 reasoning)
b85cae676953f73e3fbdf35b400ec589a27b483d
qwen/qwen3.6-27b:none__json_schema PASS 100% 213 in → 46 out (0 reasoning)
28596d7deef49c77f4f5393a7f1c6989b84bffbe
JSON Schema Structured Output
(raw) { "hash": "28596d7deef49c77f4f5393a7f1c6989b84bffbe" }
qwen/qwen3.6-27b:xhigh PASS 100% 212 in → 281 out (155 reasoning)
ef7d6162028b8e7a870500eae683d80b6eb2f46b
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 210 in → 44 out (0 reasoning)
bd540559b26bfd2bc11f045bca99b7a9aa54ed6e
JSON Schema Structured Output
(raw) { "hash": "bd540559b26bfd2bc11f045bca99b7a9aa54ed6e" }
qwen/qwen3.6-35b-a3b:high PASS 100% 214 in → 768 out (452 reasoning)
6d8f3a53c5f4fd47ca871c3c36d4a6689044789a
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 208 in → 825 out (534 reasoning)
aeb827adaacc2db7227596d72324e6bc8711a43c
JSON Schema Structured Output
(raw) { "hash": "aeb827adaacc2db7227596d72324e6bc8711a43c" }
qwen/qwen3.6-35b-a3b:low PASS 100% 212 in → 339 out (215 reasoning)
1d813389287ef94fc77e5a21dee68e775e249a57
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 207 in → 259 out (146 reasoning)
af1978abbcd4ca3e55ec3ae045c11d21d9c8ae18
JSON Schema Structured Output
(raw) { "hash": "af1978abbcd4ca3e55ec3ae045c11d21d9c8ae18" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 209 in → 338 out (199 reasoning)
c10d7df7fdbbf5b53783b12fb2033de50790348c
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 209 in → 304 out (159 reasoning)
72a04ee6f681da943b875ca098bf79fae8ca2fde
JSON Schema Structured Output
(raw) { "hash": "72a04ee6f681da943b875ca098bf79fae8ca2fde" }
qwen/qwen3.6-35b-a3b:none PASS 100% 210 in → 33 out (0 reasoning)
94b3f3eddaad4caa691f8a4dc3c42066dea62850
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 214 in → 49 out (0 reasoning)
e6ea8c86f73c59d9730c9098ba70f28b11ad255e
JSON Schema Structured Output
(raw) { "hash": "e6ea8c86f73c59d9730c9098ba70f28b11ad255e" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 212 in → 376 out (335 reasoning)
446182935d1bbd568c512246e90ace26f156aef8
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 214 in → 937 out (540 reasoning)
c1e35c74de453214841894257b06b94512f4b71d
JSON Schema Structured Output
(raw) { "hash": "c1e35c74de453214841894257b06b94512f4b71d" }
qwen/qwen3.6-flash:high PASS 100% 214 in → 811 out (769 reasoning)
1d677bbad522365629873f1d7599643712eb88d2
qwen/qwen3.6-flash:low PASS 100% 212 in → 388 out (348 reasoning)
2bb0a51057ef9cdf73c28117643945bb7a68c92a
qwen/qwen3.6-flash:medium PASS 100% 211 in → 295 out (256 reasoning)
1ee5275fe7f9b43bcde48fa82b92685d24d02ae5
qwen/qwen3.6-flash:none PASS 100% 212 in → 33 out (0 reasoning)
6cd1b5ced3f7b99f3db6b8d261c94f10b5adf0ec
qwen/qwen3.6-flash:xhigh PASS 100% 215 in → 592 out (549 reasoning)
68a68d7b01228c5d1fa6929782fa5c8685315a71
qwen/qwen3.7-flash:high PASS 100% 209 in → 419 out (381 reasoning)
6c12384afe7408ec84428e25a1fffe7ee8364bca
qwen/qwen3.7-flash:low PASS 100% 209 in → 267 out (229 reasoning)
33377ccbebc174521e1ede342cc363fbb348f80a
qwen/qwen3.7-flash:max PASS 100% 215 in → 379 out (335 reasoning)
734a012963de60285e96419268583c4231f08795
qwen/qwen3.7-flash:medium PASS 100% 212 in → 290 out (249 reasoning)
1fb0fbc8d66599df9f771383f04a15181be86782
qwen/qwen3.7-flash:none PASS 100% 210 in → 32 out (0 reasoning)
069cf5e003aebfdcc7497b1f14eb5115c5dffbb5
qwen/qwen3.7-flash:xhigh PASS 100% 215 in → 243 out (199 reasoning)
244894fb15a2483c20974c619f638d5301f9a17a
qwen/qwen3.7-max:high PASS 100% 211 in → 237 out (197 reasoning)
50a8213bdfa73070de77d240558ff59f7993bae6
qwen/qwen3.7-max:low PASS 100% 213 in → 259 out (216 reasoning)
aa223884721de77a7e58e7c1717d01594a370d1e
qwen/qwen3.7-max:medium PASS 100% 209 in → 200 out (162 reasoning)
23c82086c0aaf9aff40634d74b61f6dcea007afe
qwen/qwen3.7-max:none PASS 100% 211 in → 34 out (0 reasoning)
a8ec0441d2a7df3bb4892c3110ddc0ecbb23555d
qwen/qwen3.7-max:xhigh PASS 100% 214 in → 372 out (328 reasoning)
d13e0542f50880833b60016c1685858af5058953
qwen/qwen3.7-plus:high PASS 100% 209 in → 247 out (209 reasoning)
869bc7bbcffd31f7dc419655af2b37519ecb640e
qwen/qwen3.7-plus:low PASS 100% 211 in → 413 out (373 reasoning)
91adef8ee617f859ae945f924b5f4ce115f95540
qwen/qwen3.7-plus:medium PASS 100% 211 in → 265 out (224 reasoning)
a427d406d3d8bde9c20eed3d85227788984c9cdd
qwen/qwen3.7-plus:none PASS 100% 212 in → 35 out (0 reasoning)
d7c1274b6100ac4ea78bbf2157f0ca9d4ff1350d
qwen/qwen3.7-plus:xhigh PASS 100% 211 in → 241 out (200 reasoning)
d389280c6af381a6936c5e5c39974498daeec0c9
tencent/hy3:high PASS 100% 191 in → 145 out (81 reasoning)
6266cdb41a4cde1fbbb209a2a1a0dfdc4e09a0f8
tencent/hy3:high__json_schema PASS 100% 187 in → 292 out (210 reasoning)
cec14d2e8dd9a5468e1f124076bb6b3b1956565e
JSON Schema Structured Output
(raw) { "hash": "cec14d2e8dd9a5468e1f124076bb6b3b1956565e" }
tencent/hy3:low PASS 100% 186 in → 148 out (122 reasoning)
52df8d2f92bc0291a84920842f7029d4ce3d9997
tencent/hy3:low__json_schema PASS 100% 186 in → 168 out (106 reasoning)
942694c99823236c30589df2a1b3ac0e3a0abc9b
JSON Schema Structured Output
(raw) {"hash":"942694c99823236c30589df2a1b3ac0e3a0abc9b"}
tencent/hy3:medium PASS 100% 188 in → 230 out (144 reasoning)
8182e6f19e9eca025f6f920f6f7e2e6cb9060647
tencent/hy3:medium__json_schema PASS 100% 185 in → 149 out (85 reasoning)
f58918e8d50b45cc10662ac9e3421297f0b7c794
JSON Schema Structured Output
(raw) {"hash":"f58918e8d50b45cc10662ac9e3421297f0b7c794"}
tencent/hy3:none PASS 100% 185 in → 21 out (0 reasoning)
640d162983043bf01e994ad64a6f45636a544c30
tencent/hy3:xhigh PASS 100% 190 in → 303 out (272 reasoning)
d34c573eae9dcb2b4d6079c2031c2a9e904d9f7d
tencent/hy3:xhigh__json_schema PASS 100% 186 in → 169 out (104 reasoning)
7d5495b5bf36a6167226cec4495e609c2a00b481
JSON Schema Structured Output
(raw) { "hash": "7d5495b5bf36a6167226cec4495e609c2a00b481" }
thinkingmachines/inkling-small:high PASS 100% 189 in → 149 out (117 reasoning)
a195c2313e7bb9aa2de5dfe0a13f2ed3797e1bae
thinkingmachines/inkling-small:high__json_schema PASS 100% 185 in → 163 out (128 reasoning)
aa6f6a072cc8531545e56a62f0675122161f402f
JSON Schema Structured Output
(raw) { "hash": "aa6f6a072cc8531545e56a62f0675122161f402f" }
thinkingmachines/inkling-small:low PASS 100% 187 in → 62 out (17 reasoning)
d72f6d4cbdd3aaa2855101d71ff680956a01e4c1
thinkingmachines/inkling-small:low__json_schema PASS 100% 186 in → 102 out (68 reasoning)
036c78f7e3ba1044eea9b272424aa918214969d1
JSON Schema Structured Output
(raw) { "hash": "036c78f7e3ba1044eea9b272424aa918214969d1" }
thinkingmachines/inkling-small:max PASS 100% 187 in → 145 out (94 reasoning)
b650d38c34ae0d4f7579f108afc9a09271315aa0
thinkingmachines/inkling-small:max__json_schema PASS 100% 190 in → 150 out (111 reasoning)
33421c6f367b4b266dbbc84e0e72e61d1b7a9548
JSON Schema Structured Output
(raw) { "hash": "33421c6f367b4b266dbbc84e0e72e61d1b7a9548" }
thinkingmachines/inkling-small:medium PASS 100% 189 in → 136 out (81 reasoning)
6a1383018b555569439c5e55c9f95a92e4f8fab8
thinkingmachines/inkling-small:medium__json_schema PASS 100% 187 in → 147 out (112 reasoning)
9f9ed76e1f635912e7cc6674a1a882cbc051d828
JSON Schema Structured Output
(raw) { "hash": "9f9ed76e1f635912e7cc6674a1a882cbc051d828" }
thinkingmachines/inkling-small:none PASS 100% 186 in → 25 out (0 reasoning)
1d6796df8904a705b42ff5efbe2992e1bfadb757<|end_message|>
thinkingmachines/inkling-small:xhigh PASS 100% 187 in → 142 out (94 reasoning)
f3721a6e4f470d37a708459f4370bfc920aea7a7
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 191 in → 150 out (113 reasoning)
9b76b0c136d46a8aa4723f91acdf4e8a8e228ef6
JSON Schema Structured Output
(raw) {"hash": "9b76b0c136d46a8aa4723f91acdf4e8a8e228ef6"}
thinkingmachines/inkling:high PASS 100% 186 in → 69 out (41 reasoning)
31ea7459e6117e2817a0230c115ff031dda2c60b
thinkingmachines/inkling:high__json_schema PASS 100% 185 in → 93 out (60 reasoning)
954f4ba8b80596446512acda8815f1277a11800c
JSON Schema Structured Output
(raw) { "hash": "954f4ba8b80596446512acda8815f1277a11800c" }
thinkingmachines/inkling:low PASS 100% 188 in → 65 out (35 reasoning)
7d6fd5673b608cbda32e97e9e796672a32f7d400
thinkingmachines/inkling:low__json_schema PASS 100% 190 in → 78 out (38 reasoning)
b9f00418f2eed4b3b6ba1871d49f790d3d0978e6
JSON Schema Structured Output
(raw) { "hash": "b9f00418f2eed4b3b6ba1871d49f790d3d0978e6" }
thinkingmachines/inkling:max PASS 100% 187 in → 90 out (50 reasoning)
21255a10e75addb906659b8a6d615f782bac8e1d
thinkingmachines/inkling:max__json_schema PASS 100% 185 in → 84 out (53 reasoning)
d09f2941167d173ad77d39dea1dd9061d4ba20ac
JSON Schema Structured Output
(raw) {"hash":"d09f2941167d173ad77d39dea1dd9061d4ba20ac"}
thinkingmachines/inkling:medium PASS 100% 188 in → 69 out (27 reasoning)
3a6567793c65da1a4ed2455f2269d0dc81144af6
thinkingmachines/inkling:none PASS 100% 190 in → 29 out (0 reasoning)
4fbe2a8f89f0d18e64816e0a19247edfcfa448a3<|end_message|>
thinkingmachines/inkling:xhigh PASS 100% 185 in → 111 out (78 reasoning)
cad820d3a10f728014452ada2a1bbde1bcaa1e76
x-ai/grok-4.3:high PASS 100% 356 in → 192 out (167 reasoning)
a8c21f9085851536d64e9c38668d29ab5f8d670d
x-ai/grok-4.3:high__json_schema PASS 100% 410 in → 540 out (512 reasoning)
6731bbf0c98d7b5bb1cf5aaae90e5295e82764ca
JSON Schema Structured Output
(raw) {"hash":"6731bbf0c98d7b5bb1cf5aaae90e5295e82764ca"}
x-ai/grok-4.3:low PASS 100% 357 in → 175 out (155 reasoning)
b366d2f3132913278630397dcd1fa69dc7380ba4
x-ai/grok-4.3:low__json_schema PASS 100% 414 in → 203 out (177 reasoning)
7fcd37b70af62e5390126e544c295b7990bd3c00
JSON Schema Structured Output
(raw) {"hash":"7fcd37b70af62e5390126e544c295b7990bd3c00"}
x-ai/grok-4.3:max PASS 100% 350 in → 311 out (293 reasoning)
25cf373ba464554e9001bcc2cf5546daa591e987
x-ai/grok-4.3:max__json_schema PASS 100% 410 in → 333 out (305 reasoning)
4b6908a2749354e26e7987a48433ded0bf4b1c63
JSON Schema Structured Output
(raw) {"hash":"4b6908a2749354e26e7987a48433ded0bf4b1c63"}
x-ai/grok-4.3:medium PASS 100% 362 in → 278 out (254 reasoning)
98b59de9aa4fa069667adbd5f2fe2a3f3773708a
x-ai/grok-4.3:medium__json_schema PASS 100% 414 in → 460 out (434 reasoning)
31434730795dae3e7bd2f3429e008ea2cc350f8f
JSON Schema Structured Output
(raw) {"hash":"31434730795dae3e7bd2f3429e008ea2cc350f8f"}
x-ai/grok-4.3:none PASS 100% 351 in → 21 out (0 reasoning)
7158951b150f8336f9a032deed75846849c1cf60
x-ai/grok-4.3:none__json_schema PASS 100% 408 in → 28 out (0 reasoning)
6652bd603f9f47f271936daaee2a6f4ac106f1b8
JSON Schema Structured Output
(raw) {"hash":"6652bd603f9f47f271936daaee2a6f4ac106f1b8"}
x-ai/grok-4.3:xhigh PASS 100% 355 in → 249 out (225 reasoning)
b91e35efa1d4d4e7b349ab5078373ee4d73db916
x-ai/grok-4.3:xhigh__json_schema PASS 100% 413 in → 418 out (387 reasoning)
1c8299f26735237e6a95b0f3183cc1e8b7d4ca38
JSON Schema Structured Output
(raw) {"hash":"1c8299f26735237e6a95b0f3183cc1e8b7d4ca38"}
x-ai/grok-4.5:high PASS 100% 382 in → 201 out (175 reasoning)
bc2f6b1102dc31939c4cc1d83f585ac9b34a0da2
x-ai/grok-4.5:high__json_schema PASS 100% 443 in → 304 out (277 reasoning)
4a519a933d916a9eaa288464ce974bc43b0d91eb
JSON Schema Structured Output
(raw) {"hash":"4a519a933d916a9eaa288464ce974bc43b0d91eb"}
x-ai/grok-4.5:low PASS 100% 380 in → 106 out (82 reasoning)
d0d65d939ea7586769702fd47a6af21ab4a9b78a
x-ai/grok-4.5:low__json_schema PASS 100% 445 in → 233 out (203 reasoning)
e8f11e31c0de2a58698ee2c5feff12939deeee3c
JSON Schema Structured Output
(raw) {"hash":"e8f11e31c0de2a58698ee2c5feff12939deeee3c"}
x-ai/grok-4.5:max PASS 100% 374 in → 318 out (300 reasoning)
befdfc76647552d246878b019e22ef8445037711
x-ai/grok-4.5:max__json_schema PASS 100% 446 in → 400 out (370 reasoning)
8d882b0e4f87b89308df6572e3a1363cd7334c85
JSON Schema Structured Output
(raw) {"hash":"8d882b0e4f87b89308df6572e3a1363cd7334c85"}
x-ai/grok-4.5:medium PASS 100% 382 in → 209 out (183 reasoning)
d983a1ad3a603ab0a133b9432959c5c9a2b5de85
x-ai/grok-4.5:medium__json_schema PASS 100% 447 in → 327 out (296 reasoning)
65ce6b8b2d5f92af8ad4d35ea1929194f29a927d
JSON Schema Structured Output
(raw) {"hash":"65ce6b8b2d5f92af8ad4d35ea1929194f29a927d"}
x-ai/grok-4.5:xhigh PASS 100% 379 in → 362 out (339 reasoning)
f21fc957c13e28b498fa6bb48e9f87890f6967ae
x-ai/grok-4.5:xhigh__json_schema PASS 100% 443 in → 398 out (371 reasoning)
459b7c18bb35fbc3cd38740dccef42fb79cbf23d
JSON Schema Structured Output
(raw) {"hash":"459b7c18bb35fbc3cd38740dccef42fb79cbf23d"}
z-ai/glm-4.7-flash:high PASS 100% 178 in → 503 out (414 reasoning)
feec8ca8bbea25cbb64783638979851bddf8ad27
z-ai/glm-4.7-flash:low PASS 100% 183 in → 447 out (324 reasoning)
07820626e9a04d293b727f67e53601f2b6db9bd5
z-ai/glm-4.7-flash:low__json_schema PASS 100% 180 in → 456 out (347 reasoning)
df7b17311bc7079ce4adfd6db2cb757d005f60d0
JSON Schema Structured Output
(raw) { "hash": "df7b17311bc7079ce4adfd6db2cb757d005f60d0" }
z-ai/glm-4.7-flash:medium PASS 100% 178 in → 449 out (359 reasoning)
a57dfea1340adabe5352a9c58a0491bb05a33005
z-ai/glm-4.7-flash:none PASS 100% 182 in → 28 out (0 reasoning)
0d17971543d072a7b83744f9fe98fefb8f96c836
z-ai/glm-4.7-flash:none__json_schema PASS 100% 180 in → 35 out (0 reasoning)
d4e23644fdebfd9859bb80550637b58e5b973c81
JSON Schema Structured Output
(raw) { "hash": "d4e23644fdebfd9859bb80550637b58e5b973c81" }
z-ai/glm-4.7-flash:xhigh PASS 100% 179 in → 458 out (379 reasoning)
23385becdcb60de6ecee5120c00d9392023dd7a4
z-ai/glm-5.2:high PASS 100% 189 in → 322 out (292 reasoning)
325c56549d140a6ca86dd63b7e45e276f1aa71c3
z-ai/glm-5.2:high__json_schema PASS 100% 189 in → 140 out (106 reasoning)
94d03d8a5ce4507bd5dfa59b22be18500e2a44b8
JSON Schema Structured Output
(raw) { "hash": "94d03d8a5ce4507bd5dfa59b22be18500e2a44b8" }
z-ai/glm-5.2:low PASS 100% 193 in → 367 out (334 reasoning)
99d8b1e9090968f217c1a89e8f5e19d5b2a79524
z-ai/glm-5.2:low__json_schema PASS 100% 183 in → 183 out (100 reasoning)
41884b20bc4b3b0e00a7fc995d326af4783cfeb1
JSON Schema Structured Output
(raw) { "hash": "41884b20bc4b3b0e00a7fc995d326af4783cfeb1" }
z-ai/glm-5.2:medium PASS 100% 186 in → 243 out (217 reasoning)
ebe207caee2806083cf266d3d5b454348bfdbb44
z-ai/glm-5.2:medium__json_schema PASS 100% 186 in → 288 out (214 reasoning)
d68da821674704bfacb997b3f0b42e65e21295ba
JSON Schema Structured Output
(raw) {"hash":"d68da821674704bfacb997b3f0b42e65e21295ba"}
z-ai/glm-5.2:none PASS 100% 183 in → 28 out (0 reasoning)
123e549c0be0e61682dc208323f76f9746d3391a
z-ai/glm-5.2:none__json_schema PASS 100% 356 in → 32 out (0 reasoning)
41a33aae089a75595ca638d598f544c22a51c421
JSON Schema Structured Output
(raw) {"hash":"41a33aae089a75595ca638d598f544c22a51c421"}
z-ai/glm-5.2:xhigh PASS 100% 188 in → 343 out (315 reasoning)
ec2db461bf67c59dad2e7b1e39f2264ef78894f2
z-ai/glm-5.2:xhigh__json_schema PASS 100% 189 in → 319 out (221 reasoning)
08970a6c05dfe22fa13b420ac14f2b460c96e1b0
JSON Schema Structured Output
(raw) { "hash": "08970a6c05dfe22fa13b420ac14f2b460c96e1b0" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: f72f4e6290b29e46ea3cc9a694f16c9437f15ad2
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 225 in → 168 out (141 reasoning)
50689f3f26d0d381f6abf8eadacd037604df06c5c
Failure: Expected full hash 50689f3f26d0381f6abf8eadacd037604df06c5c, got 50689f3f26d0d381f6abf8eadacd037604df06c5c
Invalid JSON. Output: 7d9f4971468e9f793b3f870fca5f9076c5a1bb5a
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
openai/gpt-oss-20b:high__json_schema FAIL 0% 237 in → 5 out (0 reasoning)
analysis
JSON Schema Structured Output
(raw) {"hash":"analysis"}
Failure: Expected full hash 701490026be5b47a0fad802f13bb97ddc138c3cb, got analysis
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9a201827-264a-96d2-83e2-b868f5a747da","request_id":"9a201827-264a-96d2-83e2-b868f5a747da"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-99201853-cf23-9e9b-84a0-72d19038c246","request_id":"99201853-cf23-9e9b-84a0-72d19038c246"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cd43c181-d409-9b12-84c0-05b6a1fa9f41","request_id":"cd43c181-d409-9b12-84c0-05b6a1fa9f41"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a976163f-18ee-9800-9356-fed74294f9b4","request_id":"a976163f-18ee-9800-9356-fed74294f9b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7b862278-08a3-9388-a43a-d4d995073bf6","request_id":"7b862278-08a3-9388-a43a-d4d995073bf6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4ea8ee74-4eeb-984b-a31a-28abc59d8394"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3f8a101d-473b-9309-b605-861c460ebd89"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-41de9892-e219-932b-9781-e9563936021e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1470a690-76f3-95b4-b3be-373a06a2c530"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5a8db031-eb27-9953-88b6-5f3fc45f3556"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7c7855e0-138f-92fc-bcf3-1b40d983f57f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90dd1687-ac75-9f80-97ae-cffd58d046d4","request_id":"90dd1687-ac75-9f80-97ae-cffd58d046d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-553def4e-46c3-912a-9188-8de248cc4fab","request_id":"553def4e-46c3-912a-9188-8de248cc4fab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-78e270ca-05d1-9fbc-9c8c-0e0a63091f67","request_id":"78e270ca-05d1-9fbc-9c8c-0e0a63091f67"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8abf6334-5c33-96e6-8134-21f8ae547091","request_id":"8abf6334-5c33-96e6-8134-21f8ae547091"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-20f1cd78-8700-9888-948c-ab65bcc21ad8","request_id":"20f1cd78-8700-9888-948c-ab65bcc21ad8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fca3229d-dca4-9744-be53-4ee11d7b4de9","request_id":"fca3229d-dca4-9744-be53-4ee11d7b4de9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-173cc99f-c4bc-93d2-8e71-4ed584e241a6","request_id":"173cc99f-c4bc-93d2-8e71-4ed584e241a6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d4565fdf-41b7-9159-8ccb-5c6d1fe875fa","request_id":"d4565fdf-41b7-9159-8ccb-5c6d1fe875fa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-28f3e8bc-0da9-9e3c-8944-71f33fb3fea4","request_id":"28f3e8bc-0da9-9e3c-8944-71f33fb3fea4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d2928705-7e41-9965-8deb-988e71d070cf","request_id":"d2928705-7e41-9965-8deb-988e71d070cf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 166 reasoning token(s)
Invalid JSON. Output: 7fc5901a1c19e14c42c363eab48045d4e5644098{"hash": "7fc5901a1c19e14c42c363eab48045d4e5644098"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: f05fc846448056a1d123be52afb13730f425acee
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: aadf0565ac0cdc3bb6aede1bcd359c4151ce9420{"hash": "aadf0565ac0cdc3bb6aede1bcd359c4151ce9420"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 4cd6deffa52865aa3b00f2e4adea38cf03421803
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: 0a59853e9b5b610092f61c5647d0c7529a4aec2e
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)