View .gitmodules content
Tests ability to view .gitmodules file content for submodule configuration. Evaluates submodule-configuration inspection.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
View the contents of the .gitmodules file to verify the submodule configuration. Output ONLY the command to run (e.g., cat .gitmodules), nothing else.
Expected
cat .gitmodules

Scoped model quality, cost, API time, and token usage for submodule_usage/f011.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 139 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-fable-5:high__json_schema PASS 100% 345 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-fable-5:low PASS 100% 135 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-fable-5:low__json_schema PASS 100% 348 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-fable-5:max PASS 100% 138 in → 66 out (19 reasoning)
cat .gitmodules
anthropic/claude-fable-5:max__json_schema PASS 100% 349 in → 86 out (38 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-fable-5:medium PASS 100% 146 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-fable-5:medium__json_schema PASS 100% 350 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-fable-5:xhigh PASS 100% 138 in → 22 out (23 reasoning)
cat .gitmodules
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 352 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-haiku-4.5:high PASS 100% 143 in → 111 out (94 reasoning)
cat .gitmodules
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 308 in → 229 out (221 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-haiku-4.5:low PASS 100% 142 in → 97 out (83 reasoning)
cat .gitmodules
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 303 in → 416 out (403 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-haiku-4.5:medium PASS 100% 139 in → 126 out (114 reasoning)
cat .gitmodules
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 302 in → 458 out (444 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-haiku-4.5:none PASS 100% 114 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 274 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 139 in → 99 out (91 reasoning)
cat .gitmodules
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 306 in → 251 out (221 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-opus-4.6:high PASS 100% 108 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:high__json_schema PASS 100% 278 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.6:low PASS 100% 114 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:low__json_schema PASS 100% 274 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.6:max PASS 100% 111 in → 45 out (23 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:max__json_schema PASS 100% 275 in → 57 out (31 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.6:medium PASS 100% 111 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 276 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.6:none PASS 100% 115 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:none__json_schema PASS 100% 278 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.6:xhigh PASS 100% 110 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 273 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:high PASS 100% 144 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:high__json_schema PASS 100% 351 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:low PASS 100% 142 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:low__json_schema PASS 100% 351 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:max PASS 100% 149 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:max__json_schema PASS 100% 354 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:medium PASS 100% 141 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 350 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:none PASS 100% 145 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:none__json_schema PASS 100% 354 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.7:xhigh PASS 100% 147 in → 12 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 357 in → 19 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.8:high PASS 100% 140 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:high__json_schema PASS 100% 347 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-opus-4.8:low PASS 100% 133 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:low__json_schema PASS 100% 349 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-opus-4.8:max PASS 100% 139 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:max__json_schema PASS 100% 346 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-opus-4.8:medium PASS 100% 139 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 349 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-opus-4.8:none PASS 100% 135 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:none__json_schema PASS 100% 346 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-opus-4.8:xhigh PASS 100% 136 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 346 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:high PASS 100% 113 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 278 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:low PASS 100% 114 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 275 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:max PASS 100% 115 in → 44 out (23 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 273 in → 59 out (32 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:medium PASS 100% 116 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 272 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:none PASS 100% 114 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 273 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 112 in → 8 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 272 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
anthropic/claude-sonnet-5:high PASS 100% 134 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-5:high__json_schema PASS 100% 343 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-sonnet-5:low PASS 100% 140 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-5:low__json_schema PASS 100% 345 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-sonnet-5:medium PASS 100% 138 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 347 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-sonnet-5:none PASS 100% 137 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-5:none__json_schema PASS 100% 349 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
anthropic/claude-sonnet-5:xhigh PASS 100% 135 in → 10 out (0 reasoning)
cat .gitmodules
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 354 in → 17 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
arcee-ai/trinity-large-thinking:high PASS 100% 92 in → 404 out (400 reasoning)
cat .gitmodules
arcee-ai/trinity-large-thinking:low PASS 100% 96 in → 292 out (285 reasoning)
cat .gitmodules
arcee-ai/trinity-large-thinking:medium PASS 100% 92 in → 303 out (299 reasoning)
cat .gitmodules
arcee-ai/trinity-large-thinking:xhigh PASS 100% 94 in → 246 out (239 reasoning)
cat .gitmodules
arcee-ai/trinity-mini:high__json_schema PASS 100% 91 in → 278 out (313 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
arcee-ai/trinity-mini:low PASS 100% 91 in → 266 out (309 reasoning)
cat .gitmodules
arcee-ai/trinity-mini:low__json_schema PASS 100% 92 in → 329 out (353 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command":"cat .gitmodules" }
arcee-ai/trinity-mini:medium PASS 100% 94 in → 263 out (297 reasoning)
cat .gitmodules
arcee-ai/trinity-mini:medium__json_schema PASS 100% 96 in → 223 out (241 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
arcee-ai/trinity-mini:xhigh PASS 100% 97 in → 231 out (264 reasoning)
cat .gitmodules
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 95 in → 269 out (287 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command":"cat .gitmodules" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 171 in → 70 out (68 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 174 in → 65 out (53 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 91 in → 99 out (100 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 89 in → 295 out (312 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 169 in → 70 out (67 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 173 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 98 in → 79 out (73 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 91 in → 57 out (49 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 96 in → 4 out (0 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 157 in → 10 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 94 in → 61 out (59 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 96 in → 153 out (147 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash:high PASS 100% 98 in → 71 out (65 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 97 in → 79 out (65 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash:low PASS 100% 90 in → 36 out (30 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 93 in → 98 out (84 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash:medium PASS 100% 96 in → 72 out (66 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 94 in → 75 out (61 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash:none PASS 100% 90 in → 4 out (0 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 94 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 175 in → 148 out (150 reasoning)
cat .gitmodules
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 167 in → 190 out (173 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-pro:high PASS 100% 96 in → 127 out (121 reasoning)
cat .gitmodules
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 280 in → 633 out (622 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
deepseek/deepseek-v4-pro:low PASS 100% 89 in → 116 out (111 reasoning)
cat .gitmodules
deepseek/deepseek-v4-pro:medium PASS 100% 92 in → 81 out (75 reasoning)
cat .gitmodules
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 281 in → 54 out (43 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
deepseek/deepseek-v4-pro:none PASS 100% 93 in → 5 out (0 reasoning)
cat .gitmodules
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 95 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 172 in → 171 out (165 reasoning)
cat .gitmodules
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 91 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemini-3-flash-preview:high PASS 100% 110 in → 110 out (106 reasoning)
cat .gitmodules
google/gemini-3-flash-preview:high__json_schema PASS 100% 162 in → 126 out (118 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3-flash-preview:low PASS 100% 107 in → 119 out (115 reasoning)
cat .gitmodules
google/gemini-3-flash-preview:low__json_schema PASS 100% 165 in → 208 out (200 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3-flash-preview:medium PASS 100% 111 in → 64 out (60 reasoning)
cat .gitmodules
google/gemini-3-flash-preview:medium__json_schema PASS 100% 167 in → 81 out (72 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3-flash-preview:xhigh PASS 100% 107 in → 176 out (172 reasoning)
cat .gitmodules
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 109 in → 236 out (228 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 109 in → 208 out (204 reasoning)
cat .gitmodules
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 165 in → 246 out (237 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 106 in → 124 out (120 reasoning)
cat .gitmodules
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 163 in → 109 out (95 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 104 in → 93 out (89 reasoning)
cat .gitmodules
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 110 in → 151 out (142 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 108 in → 246 out (242 reasoning)
cat .gitmodules
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 160 in → 220 out (211 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.1-pro-preview:high PASS 100% 109 in → 172 out (168 reasoning)
cat .gitmodules
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 165 in → 191 out (177 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemini-3.1-pro-preview:low PASS 100% 108 in → 116 out (112 reasoning)
cat .gitmodules
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 163 in → 71 out (62 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.1-pro-preview:medium PASS 100% 107 in → 124 out (120 reasoning)
cat .gitmodules
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 163 in → 77 out (68 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 103 in → 261 out (257 reasoning)
cat .gitmodules
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 165 in → 235 out (226 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash-lite:high PASS 100% 108 in → 284 out (280 reasoning)
cat .gitmodules
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 109 in → 303 out (295 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.5-flash-lite:low PASS 100% 107 in → 4 out (0 reasoning)
cat .gitmodules
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 105 in → 9 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash-lite:max PASS 100% 108 in → 215 out (211 reasoning)
cat .gitmodules
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 166 in → 336 out (328 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.5-flash-lite:medium PASS 100% 107 in → 168 out (164 reasoning)
cat .gitmodules
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 109 in → 298 out (289 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 110 in → 266 out (262 reasoning)
cat .gitmodules
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 108 in → 400 out (392 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.5-flash:high PASS 100% 109 in → 182 out (178 reasoning)
cat .gitmodules
google/gemini-3.5-flash:high__json_schema PASS 100% 164 in → 135 out (126 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash:low PASS 100% 106 in → 155 out (151 reasoning)
cat .gitmodules
google/gemini-3.5-flash:low__json_schema PASS 100% 166 in → 153 out (144 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash:medium PASS 100% 109 in → 162 out (158 reasoning)
cat .gitmodules
google/gemini-3.5-flash:medium__json_schema PASS 100% 161 in → 167 out (158 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.5-flash:xhigh PASS 100% 108 in → 243 out (239 reasoning)
cat .gitmodules
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 165 in → 282 out (273 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemini-3.6-flash:high PASS 100% 103 in → 244 out (240 reasoning)
cat .gitmodules
google/gemini-3.6-flash:high__json_schema PASS 100% 167 in → 283 out (275 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.6-flash:low PASS 100% 107 in → 129 out (125 reasoning)
cat .gitmodules
google/gemini-3.6-flash:low__json_schema PASS 100% 110 in → 14 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemini-3.6-flash:max PASS 100% 110 in → 214 out (210 reasoning)
cat .gitmodules
google/gemini-3.6-flash:max__json_schema PASS 100% 164 in → 271 out (263 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.6-flash:medium PASS 100% 108 in → 235 out (231 reasoning)
cat .gitmodules
google/gemini-3.6-flash:medium__json_schema PASS 100% 167 in → 242 out (234 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemini-3.6-flash:xhigh PASS 100% 106 in → 213 out (209 reasoning)
cat .gitmodules
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 109 in → 426 out (418 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemma-4-26b-a4b-it:high PASS 100% 122 in → 205 out (177 reasoning)
cat .gitmodules
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 122 in → 160 out (142 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-26b-a4b-it:low PASS 100% 122 in → 160 out (151 reasoning)
cat .gitmodules
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 119 in → 243 out (212 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-26b-a4b-it:medium PASS 100% 120 in → 233 out (196 reasoning)
cat .gitmodules
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 121 in → 108 out (103 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-26b-a4b-it:none PASS 100% 120 in → 5 out (0 reasoning)
cat .gitmodules
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 116 in → 15 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 120 in → 208 out (203 reasoning)
cat .gitmodules
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 120 in → 157 out (136 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-31b-it:high PASS 100% 124 in → 97 out (88 reasoning)
cat .gitmodules
google/gemma-4-31b-it:high__json_schema PASS 100% 125 in → 98 out (81 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemma-4-31b-it:low PASS 100% 126 in → 81 out (74 reasoning)
cat .gitmodules
google/gemma-4-31b-it:low__json_schema PASS 100% 125 in → 116 out (104 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
google/gemma-4-31b-it:medium PASS 100% 127 in → 109 out (104 reasoning)
cat .gitmodules
google/gemma-4-31b-it:medium__json_schema PASS 100% 125 in → 110 out (100 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
google/gemma-4-31b-it:none PASS 100% 123 in → 5 out (0 reasoning)
cat .gitmodules
google/gemma-4-31b-it:none__json_schema PASS 100% 120 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
google/gemma-4-31b-it:xhigh PASS 100% 126 in → 93 out (89 reasoning)
cat .gitmodules
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 123 in → 136 out (117 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
ibm-granite/granite-4.1-8b PASS 100% 107 in → 5 out
cat .gitmodules
ibm-granite/granite-4.1-8b__json_schema PASS 100% 108 in → 12 out
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
inclusionai/ling-2.6-flash__json_schema PASS 100% 124 in → 15 out
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
liquid/lfm-2-24b-a2b PASS 100% 101 in → 5 out
cat .gitmodules
minimax/minimax-m2.5:high PASS 100% 125 in → 75 out (86 reasoning)
cat .gitmodules
minimax/minimax-m2.5:high__json_schema PASS 100% 125 in → 87 out (74 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m2.5:low PASS 100% 123 in → 74 out (68 reasoning)
cat .gitmodules
minimax/minimax-m2.5:low__json_schema PASS 100% 125 in → 75 out (78 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m2.5:medium PASS 100% 124 in → 78 out (83 reasoning)
cat .gitmodules
minimax/minimax-m2.5:medium__json_schema PASS 100% 137 in → 85 out (91 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m2.5:xhigh PASS 100% 124 in → 81 out (93 reasoning)
cat .gitmodules
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 134 in → 70 out (57 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m2.7:high PASS 100% 125 in → 142 out (137 reasoning)
cat .gitmodules
minimax/minimax-m2.7:low PASS 100% 121 in → 128 out (121 reasoning)
cat .gitmodules
minimax/minimax-m2.7:low__json_schema PASS 100% 255 in → 89 out (78 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
minimax/minimax-m2.7:medium PASS 100% 128 in → 209 out (232 reasoning)
cat .gitmodules
minimax/minimax-m2.7:medium__json_schema PASS 100% 250 in → 342 out (332 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
minimax/minimax-m2.7:xhigh PASS 100% 124 in → 92 out (100 reasoning)
cat .gitmodules
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 261 in → 392 out (382 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
minimax/minimax-m3:high PASS 100% 260 in → 26 out (22 reasoning)
cat .gitmodules
minimax/minimax-m3:high__json_schema PASS 100% 250 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m3:low PASS 100% 263 in → 32 out (33 reasoning)
cat .gitmodules
minimax/minimax-m3:low__json_schema PASS 100% 209 in → 12 out (11 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m3:medium PASS 100% 263 in → 48 out (49 reasoning)
cat .gitmodules
minimax/minimax-m3:medium__json_schema PASS 100% 209 in → 13 out (12 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
minimax/minimax-m3:xhigh PASS 100% 263 in → 34 out (33 reasoning)
cat .gitmodules
minimax/minimax-m3:xhigh__json_schema PASS 100% 250 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
mistralai/devstral-2512__json_schema PASS 100% 105 in → 10 out
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
mistralai/mistral-medium-3-5:high PASS 100% 114 in → 112 out (111 reasoning)
cat .gitmodules
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 106 in → 131 out (135 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
mistralai/mistral-medium-3-5:low PASS 100% 114 in → 136 out (137 reasoning)
cat .gitmodules
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 99 in → 144 out (144 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
mistralai/mistral-medium-3-5:medium PASS 100% 111 in → 94 out (95 reasoning)
cat .gitmodules
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 106 in → 73 out (66 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
mistralai/mistral-medium-3-5:none PASS 100% 112 in → 5 out (0 reasoning)
cat .gitmodules
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 102 in → 10 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 114 in → 153 out (167 reasoning)
cat .gitmodules
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 104 in → 63 out (60 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
moonshotai/kimi-k2.6:high PASS 100% 90 in → 136 out (131 reasoning)
cat .gitmodules
moonshotai/kimi-k2.6:high__json_schema PASS 100% 96 in → 397 out (179 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
moonshotai/kimi-k2.6:low PASS 100% 95 in → 148 out (142 reasoning)
cat .gitmodules
moonshotai/kimi-k2.6:low__json_schema PASS 100% 93 in → 198 out (173 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
moonshotai/kimi-k2.6:medium PASS 100% 93 in → 102 out (97 reasoning)
cat .gitmodules
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 96 in → 189 out (176 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
moonshotai/kimi-k2.6:xhigh PASS 100% 90 in → 182 out (177 reasoning)
cat .gitmodules
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 93 in → 264 out (251 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
moonshotai/kimi-k2.7-code:high PASS 100% 94 in → 43 out (37 reasoning)
cat .gitmodules
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 178 in → 64 out (53 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
moonshotai/kimi-k2.7-code:low PASS 100% 89 in → 63 out (57 reasoning)
cat .gitmodules
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 166 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
moonshotai/kimi-k2.7-code:medium PASS 100% 93 in → 32 out (31 reasoning)
cat .gitmodules
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 92 in → 97 out (86 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 95 in → 63 out (57 reasoning)
cat .gitmodules
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 177 in → 49 out (39 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
moonshotai/kimi-k3:high PASS 100% 169 in → 67 out (48 reasoning)
cat .gitmodules
moonshotai/kimi-k3:high__json_schema PASS 100% 168 in → 62 out (43 reasoning)
cat .gitmodules
moonshotai/kimi-k3:low PASS 100% 168 in → 90 out (71 reasoning)
cat .gitmodules
moonshotai/kimi-k3:low__json_schema PASS 100% 170 in → 176 out (157 reasoning)
cat .gitmodules
moonshotai/kimi-k3:max PASS 100% 169 in → 64 out (45 reasoning)
cat .gitmodules
moonshotai/kimi-k3:max__json_schema PASS 100% 165 in → 82 out (63 reasoning)
cat .gitmodules
moonshotai/kimi-k3:medium PASS 100% 168 in → 70 out (51 reasoning)
cat .gitmodules
moonshotai/kimi-k3:medium__json_schema PASS 100% 172 in → 62 out (43 reasoning)
cat .gitmodules
moonshotai/kimi-k3:xhigh PASS 100% 167 in → 95 out (76 reasoning)
cat .gitmodules
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 172 in → 67 out (48 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 115 in → 89 out (91 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 114 in → 100 out (98 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 116 in → 168 out (181 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 117 in → 178 out (141 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 118 in → 577 out (621 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 114 in → 118 out (121 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 115 in → 5 out (0 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 118 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 115 in → 115 out (124 reasoning)
cat .gitmodules
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 115 in → 249 out (247 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 115 in → 33 out (26 reasoning)
cat .gitmodules
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 112 in → 191 out (176 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 111 in → 241 out (231 reasoning)
cat .gitmodules
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 117 in → 58 out (44 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 117 in → 75 out (67 reasoning)
cat .gitmodules
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 118 in → 38 out (27 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 117 in → 5 out (0 reasoning)
cat .gitmodules
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 116 in → 13 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 118 in → 57 out (50 reasoning)
cat .gitmodules
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 116 in → 38 out (24 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
openai/gpt-5.4-mini:high PASS 100% 92 in → 50 out (40 reasoning)
cat .gitmodules
openai/gpt-5.4-mini:high__json_schema PASS 100% 129 in → 80 out (62 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-mini:low PASS 100% 94 in → 21 out (11 reasoning)
cat .gitmodules
openai/gpt-5.4-mini:low__json_schema PASS 100% 128 in → 73 out (55 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-mini:medium PASS 100% 91 in → 40 out (30 reasoning)
cat .gitmodules
openai/gpt-5.4-mini:medium__json_schema PASS 100% 127 in → 72 out (54 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-mini:none PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4-mini:none__json_schema PASS 100% 128 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-mini:xhigh PASS 100% 91 in → 95 out (85 reasoning)
cat .gitmodules
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 126 in → 173 out (155 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-nano:high PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4-nano:high__json_schema PASS 100% 126 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-nano:low PASS 100% 94 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4-nano:low__json_schema PASS 100% 130 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-nano:medium PASS 100% 88 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4-nano:medium__json_schema PASS 100% 125 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-nano:none PASS 100% 91 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4-nano:none__json_schema PASS 100% 123 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4-nano:xhigh PASS 100% 95 in → 63 out (53 reasoning)
cat .gitmodules
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 125 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4:high PASS 100% 92 in → 49 out (39 reasoning)
cat .gitmodules
openai/gpt-5.4:high__json_schema PASS 100% 128 in → 90 out (72 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4:low PASS 100% 91 in → 19 out (9 reasoning)
cat .gitmodules
openai/gpt-5.4:low__json_schema PASS 100% 127 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4:medium PASS 100% 89 in → 55 out (45 reasoning)
cat .gitmodules
openai/gpt-5.4:medium__json_schema PASS 100% 125 in → 42 out (24 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4:none PASS 100% 91 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.4:none__json_schema PASS 100% 127 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.4:xhigh PASS 100% 88 in → 85 out (75 reasoning)
cat .gitmodules
openai/gpt-5.4:xhigh__json_schema PASS 100% 131 in → 145 out (127 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.5:high PASS 100% 91 in → 59 out (49 reasoning)
cat .gitmodules
openai/gpt-5.5:high__json_schema PASS 100% 134 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.5:low PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.5:low__json_schema PASS 100% 127 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.5:medium PASS 100% 90 in → 32 out (22 reasoning)
cat .gitmodules
openai/gpt-5.5:medium__json_schema PASS 100% 125 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.5:none PASS 100% 92 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.5:none__json_schema PASS 100% 129 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.5:xhigh PASS 100% 91 in → 75 out (65 reasoning)
cat .gitmodules
openai/gpt-5.5:xhigh__json_schema PASS 100% 124 in → 223 out (205 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:high PASS 100% 95 in → 25 out (15 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:high__json_schema PASS 100% 126 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:low PASS 100% 96 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:low__json_schema PASS 100% 130 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:max PASS 100% 90 in → 56 out (46 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:max__json_schema PASS 100% 124 in → 167 out (149 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:medium PASS 100% 90 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:medium__json_schema PASS 100% 130 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:none PASS 100% 92 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:none__json_schema PASS 100% 128 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-luna:xhigh PASS 100% 92 in → 27 out (17 reasoning)
cat .gitmodules
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 127 in → 102 out (84 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:high PASS 100% 91 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:high__json_schema PASS 100% 128 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:low PASS 100% 94 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:low__json_schema PASS 100% 133 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:max PASS 100% 91 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:max__json_schema PASS 100% 123 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:medium PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:medium__json_schema PASS 100% 127 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:none PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:none__json_schema PASS 100% 125 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-sol:xhigh PASS 100% 92 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 130 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:high PASS 100% 93 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:high__json_schema PASS 100% 130 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:low PASS 100% 92 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:low__json_schema PASS 100% 125 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:max PASS 100% 92 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:max__json_schema PASS 100% 128 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:medium PASS 100% 95 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:medium__json_schema PASS 100% 126 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:none PASS 100% 95 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:none__json_schema PASS 100% 129 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-5.6-terra:xhigh PASS 100% 90 in → 8 out (0 reasoning)
cat .gitmodules
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 126 in → 16 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-oss-120b:high PASS 100% 140 in → 124 out (120 reasoning)
cat .gitmodules
openai/gpt-oss-120b:high__json_schema PASS 100% 142 in → 123 out (119 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-oss-120b:low PASS 100% 140 in → 29 out (15 reasoning)
cat .gitmodules
openai/gpt-oss-120b:low__json_schema PASS 100% 222 in → 41 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
openai/gpt-oss-120b:medium PASS 100% 152 in → 76 out (70 reasoning)
cat .gitmodules
openai/gpt-oss-120b:medium__json_schema PASS 100% 148 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
openai/gpt-oss-120b:xhigh PASS 100% 137 in → 176 out (186 reasoning)
cat .gitmodules
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 151 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
openai/gpt-oss-20b:high PASS 100% 155 in → 76 out (61 reasoning)
cat .gitmodules
openai/gpt-oss-20b:high__json_schema PASS 100% 152 in → 75 out (59 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command":"cat .gitmodules" }
openai/gpt-oss-20b:low PASS 100% 141 in → 31 out (17 reasoning)
cat .gitmodules
openai/gpt-oss-20b:low__json_schema PASS 100% 146 in → 70 out (59 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-oss-20b:medium PASS 100% 137 in → 93 out (90 reasoning)
cat .gitmodules
openai/gpt-oss-20b:medium__json_schema PASS 100% 150 in → 91 out (76 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
openai/gpt-oss-20b:xhigh PASS 100% 150 in → 61 out (57 reasoning)
cat .gitmodules
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 154 in → 234 out (250 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
poolside/laguna-m.1:high PASS 100% 117 in → 218 out (211 reasoning)
cat .gitmodules
poolside/laguna-m.1:high__json_schema PASS 100% 113 in → 223 out (207 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-m.1:low PASS 100% 112 in → 158 out (151 reasoning)
cat .gitmodules
poolside/laguna-m.1:low__json_schema PASS 100% 117 in → 202 out (191 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-m.1:medium PASS 100% 116 in → 215 out (208 reasoning)
cat .gitmodules
poolside/laguna-m.1:medium__json_schema PASS 100% 112 in → 141 out (126 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-m.1:none PASS 100% 116 in → 6 out (0 reasoning)
cat .gitmodules
poolside/laguna-m.1:none__json_schema PASS 100% 113 in → 10 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-m.1:xhigh PASS 100% 116 in → 231 out (224 reasoning)
cat .gitmodules
poolside/laguna-m.1:xhigh__json_schema PASS 100% 120 in → 227 out (216 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs-2.1:high PASS 100% 115 in → 219 out (214 reasoning)
cat .gitmodules
poolside/laguna-xs-2.1:high__json_schema PASS 100% 115 in → 253 out (242 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs-2.1:low PASS 100% 113 in → 199 out (194 reasoning)
cat .gitmodules
poolside/laguna-xs-2.1:low__json_schema PASS 100% 110 in → 275 out (264 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs-2.1:medium PASS 100% 111 in → 215 out (210 reasoning)
cat .gitmodules
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 117 in → 359 out (343 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-xs-2.1:none PASS 100% 114 in → 4 out (0 reasoning)
cat .gitmodules
poolside/laguna-xs-2.1:none__json_schema PASS 100% 113 in → 15 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-xs-2.1:xhigh PASS 100% 116 in → 227 out (222 reasoning)
cat .gitmodules
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 117 in → 116 out (105 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs.2:high PASS 100% 151 in → 47 out (40 reasoning)
cat .gitmodules
poolside/laguna-xs.2:high__json_schema PASS 100% 147 in → 82 out (71 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs.2:low PASS 100% 147 in → 96 out (89 reasoning)
cat .gitmodules
poolside/laguna-xs.2:low__json_schema PASS 100% 153 in → 71 out (55 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
poolside/laguna-xs.2:medium PASS 100% 154 in → 72 out (65 reasoning)
cat .gitmodules
poolside/laguna-xs.2:medium__json_schema PASS 100% 150 in → 200 out (189 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs.2:none PASS 100% 149 in → 6 out (0 reasoning)
cat .gitmodules
poolside/laguna-xs.2:none__json_schema PASS 100% 153 in → 10 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
poolside/laguna-xs.2:xhigh PASS 100% 146 in → 57 out (50 reasoning)
cat .gitmodules
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 150 in → 133 out (122 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
qwen/qwen3.6-27b:high PASS 100% 110 in → 624 out (624 reasoning)
cat .gitmodules
qwen/qwen3.6-27b:high__json_schema PASS 100% 109 in → 308 out (303 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-27b:low PASS 100% 112 in → 307 out (305 reasoning)
cat .gitmodules
qwen/qwen3.6-27b:medium PASS 100% 108 in → 336 out (327 reasoning)
cat .gitmodules
qwen/qwen3.6-27b:medium__json_schema PASS 100% 113 in → 249 out (240 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-27b:none PASS 100% 108 in → 5 out (0 reasoning)
cat .gitmodules
qwen/qwen3.6-27b:none__json_schema PASS 100% 114 in → 15 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-27b:xhigh PASS 100% 111 in → 242 out (244 reasoning)
cat .gitmodules
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 112 in → 583 out (1 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-35b-a3b:high PASS 100% 112 in → 781 out (775 reasoning)
cat .gitmodules
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 111 in → 363 out (368 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-35b-a3b:low PASS 100% 110 in → 637 out (628 reasoning)
cat .gitmodules
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 115 in → 291 out (282 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 114 in → 289 out (291 reasoning)
cat .gitmodules
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 110 in → 293 out (288 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-35b-a3b:none PASS 100% 116 in → 5 out (0 reasoning)
cat .gitmodules
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 114 in → 15 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 111 in → 573 out (590 reasoning)
cat .gitmodules
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 111 in → 278 out (261 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
qwen/qwen3.6-flash:high PASS 100% 109 in → 679 out (670 reasoning)
cat .gitmodules
qwen/qwen3.6-flash:low PASS 100% 113 in → 254 out (245 reasoning)
cat .gitmodules
qwen/qwen3.6-flash:medium PASS 100% 111 in → 540 out (531 reasoning)
cat .gitmodules
qwen/qwen3.6-flash:none PASS 100% 115 in → 4 out (0 reasoning)
cat .gitmodules
qwen/qwen3.6-flash:xhigh PASS 100% 108 in → 340 out (331 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:high PASS 100% 112 in → 350 out (341 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:low PASS 100% 108 in → 286 out (277 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:max PASS 100% 111 in → 273 out (264 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:medium PASS 100% 109 in → 310 out (301 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:none PASS 100% 110 in → 4 out (0 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:none__json_schema PASS 100% 111 in → 4 out (0 reasoning)
cat .gitmodules
qwen/qwen3.7-flash:xhigh PASS 100% 112 in → 335 out (326 reasoning)
cat .gitmodules
qwen/qwen3.7-max:high PASS 100% 112 in → 94 out (85 reasoning)
cat .gitmodules
qwen/qwen3.7-max:low PASS 100% 116 in → 416 out (407 reasoning)
cat .gitmodules
qwen/qwen3.7-max:medium PASS 100% 108 in → 422 out (413 reasoning)
cat .gitmodules
qwen/qwen3.7-max:none PASS 100% 113 in → 4 out (0 reasoning)
cat .gitmodules
qwen/qwen3.7-max:xhigh PASS 100% 111 in → 304 out (295 reasoning)
cat .gitmodules
qwen/qwen3.7-plus:high PASS 100% 115 in → 184 out (175 reasoning)
cat .gitmodules
qwen/qwen3.7-plus:low PASS 100% 111 in → 235 out (226 reasoning)
cat .gitmodules
qwen/qwen3.7-plus:medium PASS 100% 114 in → 213 out (204 reasoning)
cat .gitmodules
qwen/qwen3.7-plus:none PASS 100% 114 in → 4 out (0 reasoning)
cat .gitmodules
qwen/qwen3.7-plus:xhigh PASS 100% 110 in → 265 out (256 reasoning)
cat .gitmodules
tencent/hy3:high PASS 100% 101 in → 188 out (182 reasoning)
cat .gitmodules
tencent/hy3:high__json_schema PASS 100% 96 in → 89 out (79 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
tencent/hy3:low PASS 100% 99 in → 94 out (88 reasoning)
cat .gitmodules
tencent/hy3:low__json_schema PASS 100% 100 in → 162 out (167 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
tencent/hy3:medium PASS 100% 97 in → 192 out (186 reasoning)
cat .gitmodules
tencent/hy3:medium__json_schema PASS 100% 103 in → 142 out (142 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
tencent/hy3:none PASS 100% 102 in → 5 out (0 reasoning)
cat .gitmodules
tencent/hy3:none__json_schema PASS 100% 99 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
tencent/hy3:xhigh PASS 100% 101 in → 229 out (223 reasoning)
cat .gitmodules
tencent/hy3:xhigh__json_schema PASS 100% 100 in → 113 out (106 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
thinkingmachines/inkling-small:high PASS 100% 102 in → 62 out (56 reasoning)
cat .gitmodules
thinkingmachines/inkling-small:high__json_schema PASS 100% 99 in → 27 out (13 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
thinkingmachines/inkling-small:low PASS 100% 101 in → 20 out (11 reasoning)
cat .gitmodules
thinkingmachines/inkling-small:low__json_schema PASS 100% 102 in → 25 out (11 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
thinkingmachines/inkling-small:max PASS 100% 102 in → 123 out (105 reasoning)
cat .gitmodules
thinkingmachines/inkling-small:max__json_schema PASS 100% 98 in → 127 out (113 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
thinkingmachines/inkling-small:medium PASS 100% 101 in → 62 out (52 reasoning)
cat .gitmodules
thinkingmachines/inkling-small:medium__json_schema PASS 100% 102 in → 75 out (61 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
thinkingmachines/inkling-small:xhigh PASS 100% 98 in → 57 out (50 reasoning)
cat .gitmodules
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 104 in → 28 out (14 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
thinkingmachines/inkling:high PASS 100% 98 in → 52 out (46 reasoning)
cat .gitmodules
thinkingmachines/inkling:high__json_schema PASS 100% 100 in → 52 out (39 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
thinkingmachines/inkling:low PASS 100% 103 in → 26 out (16 reasoning)
cat .gitmodules
thinkingmachines/inkling:low__json_schema PASS 100% 103 in → 27 out (10 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
thinkingmachines/inkling:max PASS 100% 105 in → 57 out (46 reasoning)
cat .gitmodules
thinkingmachines/inkling:max__json_schema PASS 100% 101 in → 82 out (65 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
thinkingmachines/inkling:medium PASS 100% 101 in → 25 out (15 reasoning)
cat .gitmodules
thinkingmachines/inkling:medium__json_schema PASS 100% 100 in → 59 out (42 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
thinkingmachines/inkling:none__json_schema PASS 100% 96 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules"}
thinkingmachines/inkling:xhigh PASS 100% 95 in → 97 out (94 reasoning)
cat .gitmodules
x-ai/grok-4.3:high PASS 100% 275 in → 283 out (279 reasoning)
cat .gitmodules
x-ai/grok-4.3:high__json_schema PASS 100% 329 in → 464 out (456 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.3:low PASS 100% 276 in → 213 out (209 reasoning)
cat .gitmodules
x-ai/grok-4.3:low__json_schema PASS 100% 334 in → 367 out (358 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
x-ai/grok-4.3:max PASS 100% 270 in → 113 out (109 reasoning)
cat .gitmodules
x-ai/grok-4.3:max__json_schema PASS 100% 325 in → 358 out (349 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
x-ai/grok-4.3:medium PASS 100% 274 in → 240 out (236 reasoning)
cat .gitmodules
x-ai/grok-4.3:medium__json_schema PASS 100% 329 in → 551 out (543 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.3:none PASS 100% 265 in → 4 out (0 reasoning)
cat .gitmodules
x-ai/grok-4.3:none__json_schema PASS 100% 324 in → 8 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.3:xhigh PASS 100% 273 in → 309 out (305 reasoning)
cat .gitmodules
x-ai/grok-4.3:xhigh__json_schema PASS 100% 325 in → 408 out (400 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.5:high PASS 100% 295 in → 247 out (244 reasoning)
cat .gitmodules
x-ai/grok-4.5:high__json_schema PASS 100% 365 in → 315 out (307 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
x-ai/grok-4.5:low PASS 100% 290 in → 106 out (103 reasoning)
cat .gitmodules
x-ai/grok-4.5:low__json_schema PASS 100% 361 in → 193 out (185 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
x-ai/grok-4.5:max__json_schema PASS 100% 359 in → 370 out (362 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.5:medium PASS 100% 292 in → 183 out (180 reasoning)
cat .gitmodules
x-ai/grok-4.5:medium__json_schema PASS 100% 357 in → 292 out (284 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command":"cat .gitmodules"}
x-ai/grok-4.5:xhigh PASS 100% 293 in → 173 out (170 reasoning)
cat .gitmodules
x-ai/grok-4.5:xhigh__json_schema PASS 100% 357 in → 228 out (220 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
z-ai/glm-4.7-flash:high PASS 100% 93 in → 284 out (288 reasoning)
cat .gitmodules
z-ai/glm-4.7-flash:high__json_schema PASS 100% 94 in → 214 out (202 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
z-ai/glm-4.7-flash:low PASS 100% 91 in → 216 out (227 reasoning)
cat .gitmodules
z-ai/glm-4.7-flash:low__json_schema PASS 100% 94 in → 227 out (224 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
z-ai/glm-4.7-flash:medium PASS 100% 95 in → 221 out (222 reasoning)
cat .gitmodules
z-ai/glm-4.7-flash:none PASS 100% 93 in → 4 out (0 reasoning)
cat .gitmodules
z-ai/glm-4.7-flash:none__json_schema PASS 100% 94 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
z-ai/glm-4.7-flash:xhigh PASS 100% 94 in → 222 out (215 reasoning)
cat .gitmodules
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 95 in → 307 out (307 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) {"command": "cat .gitmodules"}
z-ai/glm-5.2:high PASS 100% 97 in → 192 out (186 reasoning)
cat .gitmodules
z-ai/glm-5.2:low PASS 100% 99 in → 202 out (196 reasoning)
cat .gitmodules
z-ai/glm-5.2:low__json_schema PASS 100% 103 in → 76 out (63 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
z-ai/glm-5.2:medium PASS 100% 103 in → 155 out (149 reasoning)
cat .gitmodules
z-ai/glm-5.2:medium__json_schema PASS 100% 100 in → 210 out (202 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
z-ai/glm-5.2:none PASS 100% 95 in → 5 out (0 reasoning)
cat .gitmodules
z-ai/glm-5.2:none__json_schema PASS 100% 95 in → 12 out (0 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
z-ai/glm-5.2:xhigh PASS 100% 100 in → 228 out (241 reasoning)
cat .gitmodules
z-ai/glm-5.2:xhigh__json_schema PASS 100% 103 in → 195 out (182 reasoning)
cat .gitmodules
JSON Schema Structured Output
(raw) { "command": "cat .gitmodules" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 93 in → 365 out (413 reasoning)
cat.gitmodules
Failure: Expected 'cat .gitmodules', got ' cat.gitmodules'
Invalid JSON. Output: cat .gitmodules
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
inclusionai/ling-2.6-flash FAIL 0% 124 in → 10 out
```sh cat .gitmodules ```
Failure: Expected 'cat .gitmodules', got '```sh cat .gitmodules ```'
Invalid JSON. Output: cat .gitmodules
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/devstral-2512 FAIL 0% 100 in → 10 out
```bash cat .gitmodules ```
Failure: Expected 'cat .gitmodules', got '```bash cat .gitmodules ```'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-17a6683a-555c-913f-b2cb-80650358411f","request_id":"17a6683a-555c-913f-b2cb-80650358411f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7d018665-e836-9ae2-a18b-3c711efb28ec","request_id":"7d018665-e836-9ae2-a18b-3c711efb28ec"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da6afab8-7907-9b31-a090-1ad5f136ed7b","request_id":"da6afab8-7907-9b31-a090-1ad5f136ed7b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-78d653fa-81e8-9024-8afe-114824452a0b","request_id":"78d653fa-81e8-9024-8afe-114824452a0b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1c8d61da-0149-99bb-bcf0-01a168ebe392","request_id":"1c8d61da-0149-99bb-bcf0-01a168ebe392"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-239b3d2b-489a-92a8-b3c8-7725512bd65b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9c642667-fe8e-9e23-b4a5-569520f7ea2f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3201b4db-f714-952b-9051-a713bf6bb703"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-56175d64-cec9-9b0c-944c-0cdb9e6ad4f9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-76763fb9-f29c-961d-af8e-a73f6f7d67f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-106582c1-c6c6-949c-9791-d58805ee6977","request_id":"106582c1-c6c6-949c-9791-d58805ee6977"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c213f2bf-d28d-9b0d-b227-33b9af74b9d8","request_id":"c213f2bf-d28d-9b0d-b227-33b9af74b9d8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a23e54f9-5cef-96e3-a5c9-8e48b146b271","request_id":"a23e54f9-5cef-96e3-a5c9-8e48b146b271"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bce7eb99-b60b-9627-bfd5-0870ce6ab501","request_id":"bce7eb99-b60b-9627-bfd5-0870ce6ab501"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7af2b577-e637-9a96-a4af-be40d987f0ce","request_id":"7af2b577-e637-9a96-a4af-be40d987f0ce"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-37fbee8a-8bf6-98d2-808c-3f6e09aa8ac4","request_id":"37fbee8a-8bf6-98d2-808c-3f6e09aa8ac4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4b52a618-69db-9ecb-a0fc-7f8e9496cb44","request_id":"4b52a618-69db-9ecb-a0fc-7f8e9496cb44"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ba63ea2c-ebaa-9011-b6e1-783d3ff7e38e","request_id":"ba63ea2c-ebaa-9011-b6e1-783d3ff7e38e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f02c4286-4e6c-9524-8db4-27eb6da0fea1","request_id":"f02c4286-4e6c-9524-8db4-27eb6da0fea1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abada170-d24f-9e0f-90fa-922a001d43c2","request_id":"abada170-d24f-9e0f-90fa-922a001d43c2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 101 in → 7 out (0 reasoning)
cat .gitmodules<|end_message|>
Failure: Expected 'cat .gitmodules', got 'cat .gitmodules<|end_message|>'
Invalid JSON. Output: cat .gitmodules{ "command": "cat .gitmodules" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 98 in → 7 out (0 reasoning)
cat .gitmodules<|end_message|>
Failure: Expected 'cat .gitmodules', got 'cat .gitmodules<|end_message|>'
Invalid JSON. Output: cat .gitmodules
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
x-ai/grok-4.5:max FAIL 0% 295 in → 190 out (183 reasoning)
``` cat .gitmodules ```
Failure: Expected 'cat .gitmodules', got '``` cat .gitmodules ```'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: cat .gitmodules
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)