List submodules
Tests ability to list all submodules in a repository. Evaluates submodule-listing command knowledge.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Run the command to list all configured submodules. Output ONLY the git command, nothing else.
Expected
git submodule

Scoped model quality, cost, API time, and token usage for submodule_usage/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 113 in → 120 out (83 reasoning)
git submodule status
anthropic/claude-fable-5:high__json_schema PASS 100% 321 in → 76 out (67 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:low PASS 100% 112 in → 62 out (52 reasoning)
git submodule status
anthropic/claude-fable-5:low__json_schema PASS 100% 323 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:max PASS 100% 109 in → 429 out (79 reasoning)
``` git submodule status ```
anthropic/claude-fable-5:max__json_schema PASS 100% 324 in → 303 out (85 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:medium PASS 100% 112 in → 63 out (45 reasoning)
git submodule status
anthropic/claude-fable-5:medium__json_schema PASS 100% 322 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:xhigh PASS 100% 107 in → 217 out (86 reasoning)
``` git submodule status ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 323 in → 73 out (64 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-opus-4.6:high PASS 100% 94 in → 12 out (0 reasoning)
``` git submodule ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 259 in → 38 out (13 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.6:low PASS 100% 96 in → 12 out (0 reasoning)
``` git submodule status ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 257 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.6:max PASS 100% 93 in → 12 out (0 reasoning)
``` git submodule ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 258 in → 59 out (33 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.6:medium PASS 100% 91 in → 12 out (0 reasoning)
``` git submodule status ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 260 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.6:none PASS 100% 95 in → 12 out (0 reasoning)
``` git submodule status ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 258 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.6:xhigh PASS 100% 95 in → 12 out (0 reasoning)
``` git submodule ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 257 in → 37 out (13 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-opus-4.7:high PASS 100% 113 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:high__json_schema PASS 100% 329 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:low PASS 100% 120 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:low__json_schema PASS 100% 328 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:max__json_schema PASS 100% 327 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:medium PASS 100% 116 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 324 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:none PASS 100% 121 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:none__json_schema PASS 100% 323 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:xhigh PASS 100% 114 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 330 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 325 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:low__json_schema PASS 100% 324 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 328 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 321 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 317 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 256 in → 197 out (177 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-sonnet-4.6:low PASS 100% 93 in → 11 out (0 reasoning)
``` git submodule ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 260 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-sonnet-4.6:max PASS 100% 94 in → 211 out (199 reasoning)
``` git submodule ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 257 in → 238 out (207 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-sonnet-4.6:medium PASS 100% 92 in → 11 out (0 reasoning)
``` git submodule ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 253 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-sonnet-4.6:none PASS 100% 96 in → 12 out (0 reasoning)
``` git submodule status ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 252 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 258 in → 177 out (146 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
anthropic/claude-sonnet-5:high PASS 100% 116 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-sonnet-5:high__json_schema PASS 100% 323 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:low PASS 100% 112 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-sonnet-5:low__json_schema PASS 100% 322 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:medium PASS 100% 113 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 322 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-5:none PASS 100% 113 in → 11 out (0 reasoning)
``` git submodule status ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 322 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 321 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
arcee-ai/trinity-large-thinking:high PASS 100% 82 in → 561 out (556 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:low PASS 100% 79 in → 477 out (472 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:medium PASS 100% 77 in → 405 out (399 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:xhigh PASS 100% 84 in → 577 out (573 reasoning)
git submodule status
arcee-ai/trinity-mini:medium PASS 100% 79 in → 496 out (583 reasoning)
git submodule status
arcee-ai/trinity-mini:xhigh PASS 100% 84 in → 461 out (546 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:high PASS 100% 84 in → 435 out (429 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 81 in → 311 out (325 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 78 in → 361 out (356 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 81 in → 207 out (211 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 158 in → 1,351 out (1,421 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 158 in → 654 out (654 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 78 in → 808 out (803 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 82 in → 177 out (164 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 76 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 76 in → 356 out (398 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 168 in → 675 out (739 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:high PASS 100% 80 in → 353 out (389 reasoning)
git submodule status
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 75 in → 271 out (242 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:low PASS 100% 81 in → 389 out (359 reasoning)
git submodule status
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 76 in → 187 out (199 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:medium PASS 100% 77 in → 175 out (170 reasoning)
git submodule status
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 79 in → 264 out (278 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:none PASS 100% 80 in → 9 out (0 reasoning)
```bash git submodule status ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 78 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 157 in → 368 out (413 reasoning)
git submodule status
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 341 in → 312 out (301 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
deepseek/deepseek-v4-pro:high PASS 100% 76 in → 374 out (369 reasoning)
git submodule
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 261 in → 160 out (149 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
deepseek/deepseek-v4-pro:low PASS 100% 75 in → 583 out (577 reasoning)
git submodule status
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 268 in → 248 out (238 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command": "git submodule"}
deepseek/deepseek-v4-pro:medium PASS 100% 81 in → 411 out (406 reasoning)
git submodule
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 266 in → 282 out (272 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
deepseek/deepseek-v4-pro:none PASS 100% 77 in → 5 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 73 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 159 in → 630 out (624 reasoning)
git submodule status
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 343 in → 950 out (939 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3-flash-preview:high PASS 100% 89 in → 279 out (276 reasoning)
git submodule status
google/gemini-3-flash-preview:high__json_schema PASS 100% 149 in → 198 out (190 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3-flash-preview:low PASS 100% 94 in → 286 out (283 reasoning)
git submodule status
google/gemini-3-flash-preview:low__json_schema PASS 100% 147 in → 273 out (266 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3-flash-preview:medium PASS 100% 91 in → 607 out (604 reasoning)
git submodule status
google/gemini-3-flash-preview:medium__json_schema PASS 100% 152 in → 351 out (344 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3-flash-preview:xhigh PASS 100% 94 in → 260 out (257 reasoning)
git submodule status
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 148 in → 293 out (286 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 90 in → 293 out (291 reasoning)
git submodule
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 96 in → 227 out (219 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 91 in → 124 out (121 reasoning)
git submodule status
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 150 in → 117 out (109 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 95 in → 116 out (114 reasoning)
git submodule
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 94 in → 127 out (119 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 92 in → 372 out (370 reasoning)
git submodule
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 92 in → 180 out (168 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
google/gemini-3.1-pro-preview:high PASS 100% 95 in → 568 out (566 reasoning)
git submodule
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 148 in → 319 out (306 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.1-pro-preview:low PASS 100% 92 in → 260 out (257 reasoning)
git submodule status
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 146 in → 239 out (231 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-pro-preview:medium PASS 100% 92 in → 333 out (330 reasoning)
git submodule status
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 151 in → 101 out (94 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command": "git submodule"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 93 in → 529 out (527 reasoning)
git submodule
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 149 in → 504 out (497 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash-lite:high PASS 100% 93 in → 313 out (311 reasoning)
git submodule
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 93 in → 369 out (363 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
google/gemini-3.5-flash-lite:low PASS 100% 95 in → 3 out (0 reasoning)
git submodule status
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 93 in → 8 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash-lite:max PASS 100% 90 in → 411 out (409 reasoning)
git submodule
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 96 in → 383 out (375 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash-lite:medium PASS 100% 92 in → 193 out (191 reasoning)
git submodule
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 149 in → 327 out (315 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 90 in → 333 out (331 reasoning)
git submodule
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 94 in → 342 out (336 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
google/gemini-3.5-flash:high PASS 100% 93 in → 585 out (582 reasoning)
git submodule status
google/gemini-3.5-flash:high__json_schema PASS 100% 150 in → 362 out (354 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash:low PASS 100% 92 in → 184 out (181 reasoning)
git submodule status
google/gemini-3.5-flash:low__json_schema PASS 100% 148 in → 135 out (127 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash:medium PASS 100% 92 in → 755 out (753 reasoning)
git submodule
google/gemini-3.5-flash:medium__json_schema PASS 100% 147 in → 338 out (330 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash:xhigh PASS 100% 95 in → 696 out (693 reasoning)
git submodule status
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 150 in → 308 out (300 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.6-flash:high PASS 100% 90 in → 435 out (432 reasoning)
git submodule status
google/gemini-3.6-flash:high__json_schema PASS 100% 152 in → 719 out (712 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:low PASS 100% 91 in → 299 out (296 reasoning)
git submodule status
google/gemini-3.6-flash:low__json_schema PASS 100% 149 in → 7 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:max PASS 100% 95 in → 829 out (826 reasoning)
git submodule status
google/gemini-3.6-flash:max__json_schema PASS 100% 96 in → 752 out (745 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:medium PASS 100% 96 in → 482 out (479 reasoning)
git submodule status
google/gemini-3.6-flash:medium__json_schema PASS 100% 90 in → 384 out (377 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:xhigh PASS 100% 89 in → 812 out (809 reasoning)
git submodule status
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 146 in → 435 out (428 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemma-4-26b-a4b-it:high PASS 100% 110 in → 660 out (656 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 109 in → 605 out (587 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:low PASS 100% 106 in → 536 out (553 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 109 in → 393 out (392 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:medium PASS 100% 106 in → 835 out (836 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 106 in → 629 out (608 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:none PASS 100% 105 in → 4 out (0 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 101 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 109 in → 667 out (663 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 106 in → 545 out (510 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:high PASS 100% 109 in → 434 out (447 reasoning)
git submodule status
google/gemma-4-31b-it:high__json_schema PASS 100% 106 in → 339 out (338 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:low PASS 100% 107 in → 337 out (354 reasoning)
git submodule status
google/gemma-4-31b-it:low__json_schema PASS 100% 112 in → 201 out (197 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemma-4-31b-it:medium PASS 100% 105 in → 206 out (215 reasoning)
git submodule status
google/gemma-4-31b-it:medium__json_schema PASS 100% 104 in → 315 out (327 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemma-4-31b-it:none PASS 100% 105 in → 4 out (0 reasoning)
git submodule status
google/gemma-4-31b-it:none__json_schema PASS 100% 105 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:xhigh PASS 100% 105 in → 275 out (282 reasoning)
git submodule status
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 110 in → 263 out (262 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
inclusionai/ling-2.6-flash PASS 100% 114 in → 9 out
``` git submodule status ```
inclusionai/ling-2.6-flash__json_schema PASS 100% 113 in → 15 out
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
liquid/lfm-2-24b-a2b PASS 100% 86 in → 5 out
git submodule status
minimax/minimax-m2.5:high PASS 100% 111 in → 131 out (167 reasoning)
git submodule status
minimax/minimax-m2.5:high__json_schema PASS 100% 121 in → 77 out (66 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:low PASS 100% 110 in → 171 out (167 reasoning)
git submodule status
minimax/minimax-m2.5:low__json_schema PASS 100% 121 in → 77 out (66 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:medium PASS 100% 115 in → 91 out (125 reasoning)
git submodule status
minimax/minimax-m2.5:medium__json_schema PASS 100% 108 in → 292 out (354 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:xhigh PASS 100% 109 in → 81 out (107 reasoning)
git submodule status
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 120 in → 139 out (166 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.7:high PASS 100% 114 in → 170 out (166 reasoning)
git submodule status
minimax/minimax-m2.7:low PASS 100% 111 in → 553 out (548 reasoning)
git submodule status
minimax/minimax-m2.7:low__json_schema PASS 100% 242 in → 406 out (397 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
minimax/minimax-m2.7:medium PASS 100% 115 in → 275 out (271 reasoning)
git submodule status
minimax/minimax-m2.7:medium__json_schema PASS 100% 243 in → 493 out (484 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
minimax/minimax-m2.7:xhigh PASS 100% 108 in → 534 out (529 reasoning)
git submodule status
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 110 in → 11 out (11 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:high__json_schema PASS 100% 232 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:low PASS 100% 248 in → 217 out (266 reasoning)
git submodule status
minimax/minimax-m3:low__json_schema PASS 100% 194 in → 11 out (10 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:medium PASS 100% 241 in → 204 out (234 reasoning)
git submodule status
minimax/minimax-m3:medium__json_schema PASS 100% 235 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:xhigh PASS 100% 248 in → 168 out (192 reasoning)
git submodule status
minimax/minimax-m3:xhigh__json_schema PASS 100% 234 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/devstral-2512 PASS 100% 87 in → 10 out
```bash git submodule status ```
mistralai/devstral-2512__json_schema PASS 100% 91 in → 10 out
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:high PASS 100% 99 in → 501 out (569 reasoning)
git submodule status
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 86 in → 726 out (803 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:low PASS 100% 101 in → 1,213 out (1,326 reasoning)
git submodule status
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 87 in → 106 out (102 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:medium PASS 100% 98 in → 1,668 out (1,826 reasoning)
git submodule status
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 88 in → 715 out (802 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:none PASS 100% 99 in → 5 out (0 reasoning)
git submodule status
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 88 in → 10 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 98 in → 1,468 out (1,623 reasoning)
git submodule status
moonshotai/kimi-k2.6:high PASS 100% 76 in → 1,189 out (1,400 reasoning)
git submodule status
moonshotai/kimi-k2.6:high__json_schema PASS 100% 263 in → 601 out (592 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
moonshotai/kimi-k2.6:low PASS 100% 82 in → 1,150 out (1,146 reasoning)
git submodule status
moonshotai/kimi-k2.6:low__json_schema PASS 100% 81 in → 818 out (675 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k2.6:medium PASS 100% 81 in → 1,009 out (1,004 reasoning)
git submodule status
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 82 in → 757 out (745 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k2.6:xhigh PASS 100% 74 in → 698 out (831 reasoning)
git submodule status
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 73 in → 1,332 out (1,253 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k2.7-code:high PASS 100% 79 in → 215 out (210 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 160 in → 185 out (176 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
moonshotai/kimi-k2.7-code:medium PASS 100% 80 in → 84 out (80 reasoning)
git submodule
moonshotai/kimi-k2.7-code:xhigh PASS 100% 78 in → 354 out (410 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 79 in → 273 out (311 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k3:high PASS 100% 154 in → 144 out (126 reasoning)
git submodule status
moonshotai/kimi-k3:high__json_schema PASS 100% 151 in → 154 out (136 reasoning)
git submodule status
moonshotai/kimi-k3:low PASS 100% 152 in → 103 out (85 reasoning)
git submodule status
moonshotai/kimi-k3:low__json_schema PASS 100% 155 in → 371 out (350 reasoning)
``` git submodule status ```
moonshotai/kimi-k3:max__json_schema PASS 100% 153 in → 242 out (224 reasoning)
git submodule status
moonshotai/kimi-k3:medium PASS 100% 156 in → 269 out (251 reasoning)
git submodule status
moonshotai/kimi-k3:medium__json_schema PASS 100% 251 in → 294 out (271 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
moonshotai/kimi-k3:xhigh PASS 100% 155 in → 222 out (204 reasoning)
git submodule status
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 156 in → 182 out (164 reasoning)
git submodule status
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 99 in → 1,144 out (1,346 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 100 in → 252 out (282 reasoning)
git submodule status
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 102 in → 603 out (659 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 98 in → 379 out (435 reasoning)
git submodule status
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 100 in → 294 out (318 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 102 in → 5 out (0 reasoning)
git submodule status
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 100 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 100 in → 139 out (148 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 103 in → 182 out (161 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 101 in → 248 out (220 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 101 in → 109 out (98 reasoning)
git submodule
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 103 in → 236 out (206 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 96 in → 183 out (220 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 101 in → 185 out (161 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 98 in → 5 out (0 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 99 in → 11 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 99 in → 265 out (241 reasoning)
git submodule
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 100 in → 222 out (196 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
openai/gpt-5.4-mini:high PASS 100% 74 in → 526 out (516 reasoning)
git submodule status
openai/gpt-5.4-mini:high__json_schema PASS 100% 108 in → 196 out (178 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4-mini:low PASS 100% 81 in → 155 out (145 reasoning)
git submodule status
openai/gpt-5.4-mini:low__json_schema PASS 100% 114 in → 59 out (41 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4-mini:medium PASS 100% 74 in → 557 out (547 reasoning)
git submodule status
openai/gpt-5.4-mini:medium__json_schema PASS 100% 114 in → 286 out (268 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4-mini:xhigh PASS 100% 72 in → 751 out (741 reasoning)
git submodule status
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 108 in → 5,797 out (5,779 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4-nano:none PASS 100% 76 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.4-nano:xhigh PASS 100% 75 in → 423 out (413 reasoning)
git submodule status
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 114 in → 837 out (819 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4:high PASS 100% 77 in → 526 out (516 reasoning)
git submodule status
openai/gpt-5.4:high__json_schema PASS 100% 112 in → 2,570 out (2,552 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4:low PASS 100% 72 in → 76 out (66 reasoning)
git submodule status
openai/gpt-5.4:low__json_schema PASS 100% 112 in → 109 out (91 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4:medium PASS 100% 75 in → 416 out (406 reasoning)
git submodule status
openai/gpt-5.4:medium__json_schema PASS 100% 113 in → 193 out (175 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4:none PASS 100% 80 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.4:none__json_schema PASS 100% 113 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4:xhigh PASS 100% 75 in → 4,434 out (4,424 reasoning)
git submodule status
openai/gpt-5.4:xhigh__json_schema PASS 100% 107 in → 2,854 out (2,836 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:high PASS 100% 75 in → 182 out (172 reasoning)
git submodule status
openai/gpt-5.5:high__json_schema PASS 100% 114 in → 521 out (503 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:low PASS 100% 77 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.5:low__json_schema PASS 100% 112 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:medium PASS 100% 79 in → 151 out (141 reasoning)
git submodule status
openai/gpt-5.5:medium__json_schema PASS 100% 117 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:none PASS 100% 73 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.5:none__json_schema PASS 100% 111 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:xhigh PASS 100% 77 in → 487 out (477 reasoning)
git submodule status
openai/gpt-5.5:xhigh__json_schema PASS 100% 111 in → 478 out (460 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:high PASS 100% 76 in → 322 out (312 reasoning)
git submodule status
openai/gpt-5.6-luna:high__json_schema PASS 100% 112 in → 253 out (235 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:low PASS 100% 79 in → 141 out (131 reasoning)
git submodule status
openai/gpt-5.6-luna:max PASS 100% 77 in → 401 out (391 reasoning)
git submodule status
openai/gpt-5.6-luna:max__json_schema PASS 100% 108 in → 534 out (516 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:medium PASS 100% 78 in → 317 out (307 reasoning)
git submodule status
openai/gpt-5.6-luna:medium__json_schema PASS 100% 112 in → 92 out (74 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:none PASS 100% 75 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-luna:none__json_schema PASS 100% 108 in → 15 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
openai/gpt-5.6-luna:xhigh PASS 100% 79 in → 354 out (344 reasoning)
git submodule status
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 111 in → 316 out (299 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command":"git submodule"}
openai/gpt-5.6-sol:high PASS 100% 76 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:high__json_schema PASS 100% 116 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:low PASS 100% 75 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:low__json_schema PASS 100% 107 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:max PASS 100% 76 in → 173 out (163 reasoning)
git submodule status
openai/gpt-5.6-sol:max__json_schema PASS 100% 109 in → 168 out (150 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:medium PASS 100% 80 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:medium__json_schema PASS 100% 108 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:none PASS 100% 76 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:none__json_schema PASS 100% 109 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:xhigh PASS 100% 79 in → 68 out (58 reasoning)
git submodule status
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 112 in → 234 out (216 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:high PASS 100% 78 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-terra:high__json_schema PASS 100% 115 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:low PASS 100% 74 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-terra:low__json_schema PASS 100% 114 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:max PASS 100% 78 in → 209 out (199 reasoning)
git submodule status
openai/gpt-5.6-terra:max__json_schema PASS 100% 115 in → 119 out (101 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:medium__json_schema PASS 100% 111 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:none PASS 100% 75 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-terra:none__json_schema PASS 100% 114 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:xhigh PASS 100% 79 in → 79 out (69 reasoning)
git submodule status
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 110 in → 81 out (63 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-120b:high PASS 100% 124 in → 525 out (575 reasoning)
git submodule status
openai/gpt-oss-120b:low PASS 100% 126 in → 95 out (84 reasoning)
git submodule status
openai/gpt-oss-120b:medium PASS 100% 125 in → 290 out (297 reasoning)
git submodule status
openai/gpt-oss-120b:xhigh PASS 100% 134 in → 2,076 out (2,328 reasoning)
git submodule status
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 137 in → 601 out (556 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-20b:high PASS 100% 140 in → 725 out (680 reasoning)
git submodule status
openai/gpt-oss-20b:high__json_schema PASS 100% 137 in → 404 out (363 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
openai/gpt-oss-20b:low PASS 100% 140 in → 22 out (9 reasoning)
git submodule status
openai/gpt-oss-20b:low__json_schema PASS 100% 139 in → 57 out (46 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-20b:medium PASS 100% 135 in → 146 out (144 reasoning)
git submodule status
openai/gpt-oss-20b:medium__json_schema PASS 100% 136 in → 10 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-20b:xhigh PASS 100% 141 in → 222 out (225 reasoning)
git submodule status
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 139 in → 509 out (546 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
poolside/laguna-m.1:high PASS 100% 99 in → 536 out (531 reasoning)
git submodule
poolside/laguna-m.1:high__json_schema PASS 100% 100 in → 182 out (172 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-m.1:low PASS 100% 96 in → 441 out (435 reasoning)
git submodule status
poolside/laguna-m.1:low__json_schema PASS 100% 103 in → 225 out (215 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-m.1:medium PASS 100% 101 in → 370 out (364 reasoning)
git submodule status
poolside/laguna-m.1:medium__json_schema PASS 100% 98 in → 534 out (524 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-m.1:none PASS 100% 101 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-m.1:none__json_schema PASS 100% 102 in → 8 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
poolside/laguna-m.1:xhigh PASS 100% 102 in → 812 out (806 reasoning)
git submodule status
poolside/laguna-m.1:xhigh__json_schema PASS 100% 101 in → 579 out (569 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs-2.1:high PASS 100% 102 in → 378 out (374 reasoning)
git submodule status
poolside/laguna-xs-2.1:high__json_schema PASS 100% 99 in → 1,242 out (1,232 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs-2.1:low PASS 100% 103 in → 319 out (315 reasoning)
git submodule status
poolside/laguna-xs-2.1:low__json_schema PASS 100% 96 in → 302 out (287 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:medium PASS 100% 101 in → 248 out (244 reasoning)
git submodule status
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 99 in → 380 out (365 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:none PASS 100% 101 in → 3 out (0 reasoning)
git submodule status
poolside/laguna-xs-2.1:none__json_schema PASS 100% 102 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:xhigh PASS 100% 99 in → 475 out (471 reasoning)
git submodule status
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 102 in → 938 out (924 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs.2:high PASS 100% 137 in → 125 out (120 reasoning)
git submodule
poolside/laguna-xs.2:high__json_schema PASS 100% 136 in → 756 out (746 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:low PASS 100% 134 in → 231 out (225 reasoning)
git submodule status
poolside/laguna-xs.2:low__json_schema PASS 100% 139 in → 67 out (52 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs.2:medium PASS 100% 138 in → 1,621 out (1,615 reasoning)
git submodule status
poolside/laguna-xs.2:medium__json_schema PASS 100% 134 in → 1,133 out (1,123 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:none PASS 100% 135 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-xs.2:none__json_schema PASS 100% 140 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:xhigh PASS 100% 134 in → 386 out (381 reasoning)
git submodule
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 133 in → 530 out (515 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:high PASS 100% 97 in → 1,578 out (1,677 reasoning)
git submodule
qwen/qwen3.6-27b:low PASS 100% 96 in → 1,001 out (1,069 reasoning)
git submodule status
qwen/qwen3.6-27b:low__json_schema PASS 100% 100 in → 1,102 out (1,169 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:medium PASS 100% 97 in → 1,551 out (1,636 reasoning)
git submodule status
qwen/qwen3.6-27b:medium__json_schema PASS 100% 98 in → 173 out (186 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:none PASS 100% 97 in → 4 out (0 reasoning)
git submodule status
qwen/qwen3.6-27b:none__json_schema PASS 100% 100 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:xhigh PASS 100% 97 in → 1,385 out (1,377 reasoning)
git submodule status
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 99 in → 1,283 out (1 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:high PASS 100% 100 in → 2,074 out (2,158 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 94 in → 1,943 out (1,993 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:low PASS 100% 95 in → 1,396 out (1,450 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 97 in → 1,989 out (2,039 reasoning)
git submodule
JSON Schema Structured Output
(raw) { "command": "git submodule" }
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 99 in → 1,310 out (1,416 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:none PASS 100% 99 in → 4 out (0 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 100 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 95 in → 1,342 out (1,415 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 99 in → 1,126 out (1,208 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-flash:high PASS 100% 100 in → 1,856 out (1,848 reasoning)
git submodule status
qwen/qwen3.6-flash:low PASS 100% 97 in → 1,661 out (1,653 reasoning)
git submodule status
qwen/qwen3.6-flash:medium PASS 100% 93 in → 2,986 out (2,978 reasoning)
git submodule status
qwen/qwen3.6-flash:none PASS 100% 100 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.6-flash:xhigh PASS 100% 98 in → 1,640 out (1,632 reasoning)
git submodule status
qwen/qwen3.7-flash:high PASS 100% 98 in → 1,711 out (1,703 reasoning)
git submodule status
qwen/qwen3.7-flash:low PASS 100% 98 in → 1,480 out (1,472 reasoning)
git submodule status
qwen/qwen3.7-flash:medium PASS 100% 98 in → 1,264 out (1,256 reasoning)
git submodule status
qwen/qwen3.7-flash:none PASS 100% 102 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-flash:xhigh PASS 100% 98 in → 2,025 out (2,017 reasoning)
git submodule status
qwen/qwen3.7-max:high PASS 100% 99 in → 727 out (719 reasoning)
git submodule status
qwen/qwen3.7-max:low PASS 100% 97 in → 788 out (780 reasoning)
git submodule status
qwen/qwen3.7-max:medium PASS 100% 100 in → 595 out (587 reasoning)
git submodule status
qwen/qwen3.7-max:none PASS 100% 98 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-max:xhigh PASS 100% 101 in → 821 out (813 reasoning)
git submodule status
qwen/qwen3.7-plus:high PASS 100% 97 in → 1,004 out (996 reasoning)
git submodule status
qwen/qwen3.7-plus:low PASS 100% 99 in → 670 out (662 reasoning)
git submodule status
qwen/qwen3.7-plus:medium PASS 100% 94 in → 920 out (912 reasoning)
git submodule status
qwen/qwen3.7-plus:none PASS 100% 102 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-plus:xhigh PASS 100% 95 in → 793 out (786 reasoning)
git submodule
tencent/hy3:high PASS 100% 85 in → 1,045 out (1,127 reasoning)
git submodule status
tencent/hy3:high__json_schema PASS 100% 86 in → 829 out (892 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:low PASS 100% 87 in → 1,008 out (1,003 reasoning)
git submodule status
tencent/hy3:low__json_schema PASS 100% 84 in → 944 out (985 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:medium PASS 100% 89 in → 722 out (809 reasoning)
git submodule status
tencent/hy3:medium__json_schema PASS 100% 92 in → 896 out (973 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:none PASS 100% 88 in → 4 out (0 reasoning)
git submodule status
tencent/hy3:none__json_schema PASS 100% 89 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:xhigh PASS 100% 85 in → 662 out (657 reasoning)
git submodule status
tencent/hy3:xhigh__json_schema PASS 100% 87 in → 1,566 out (1,701 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:high PASS 100% 84 in → 536 out (578 reasoning)
git submodule status
thinkingmachines/inkling-small:high__json_schema PASS 100% 81 in → 381 out (364 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:low PASS 100% 88 in → 102 out (107 reasoning)
git submodule status
thinkingmachines/inkling-small:low__json_schema PASS 100% 84 in → 120 out (104 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:max PASS 100% 88 in → 679 out (669 reasoning)
git submodule status
thinkingmachines/inkling-small:max__json_schema PASS 100% 87 in → 592 out (578 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
thinkingmachines/inkling-small:medium PASS 100% 90 in → 348 out (338 reasoning)
git submodule status
thinkingmachines/inkling-small:medium__json_schema PASS 100% 85 in → 288 out (272 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:xhigh PASS 100% 90 in → 319 out (354 reasoning)
git submodule status
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 88 in → 146 out (161 reasoning)
git submodule status
thinkingmachines/inkling:high PASS 100% 85 in → 343 out (382 reasoning)
git submodule status
thinkingmachines/inkling:high__json_schema PASS 100% 82 in → 1,539 out (1,526 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
thinkingmachines/inkling:low PASS 100% 82 in → 23 out (14 reasoning)
git submodule
thinkingmachines/inkling:max PASS 100% 85 in → 3,097 out (3,087 reasoning)
git submodule status
thinkingmachines/inkling:max__json_schema PASS 100% 85 in → 1,017 out (1,001 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling:medium PASS 100% 85 in → 365 out (410 reasoning)
git submodule status
thinkingmachines/inkling:medium__json_schema PASS 100% 87 in → 526 out (509 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling:none__json_schema PASS 100% 85 in → 12 out (0 reasoning)
git submodule
JSON Schema Structured Output
(raw) {"command": "git submodule"}
thinkingmachines/inkling:xhigh PASS 100% 84 in → 1,014 out (1,147 reasoning)
git submodule status
thinkingmachines/inkling:xhigh__json_schema PASS 100% 82 in → 718 out (702 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
x-ai/grok-4.3:high PASS 100% 256 in → 365 out (357 reasoning)
```bash git submodule status ```
x-ai/grok-4.3:high__json_schema PASS 100% 311 in → 710 out (703 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:low PASS 100% 262 in → 345 out (342 reasoning)
git submodule status
x-ai/grok-4.3:low__json_schema PASS 100% 317 in → 405 out (394 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
x-ai/grok-4.3:max__json_schema PASS 100% 310 in → 791 out (780 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
x-ai/grok-4.3:medium PASS 100% 259 in → 1,589 out (1,586 reasoning)
git submodule status
x-ai/grok-4.3:medium__json_schema PASS 100% 318 in → 865 out (858 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:none PASS 100% 256 in → 3 out (0 reasoning)
git submodule status
x-ai/grok-4.3:none__json_schema PASS 100% 308 in → 7 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:xhigh__json_schema PASS 100% 316 in → 592 out (585 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.5:high PASS 100% 282 in → 405 out (402 reasoning)
git submodule status
x-ai/grok-4.5:high__json_schema PASS 100% 347 in → 525 out (517 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:low PASS 100% 284 in → 200 out (193 reasoning)
``` git submodule status ```
x-ai/grok-4.5:low__json_schema PASS 100% 344 in → 321 out (313 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:max PASS 100% 284 in → 201 out (194 reasoning)
``` git submodule status ```
x-ai/grok-4.5:max__json_schema PASS 100% 346 in → 438 out (430 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:medium PASS 100% 280 in → 369 out (366 reasoning)
git submodule status
x-ai/grok-4.5:medium__json_schema PASS 100% 349 in → 1,005 out (997 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.5:xhigh PASS 100% 277 in → 274 out (271 reasoning)
git submodule status
x-ai/grok-4.5:xhigh__json_schema PASS 100% 350 in → 525 out (517 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
z-ai/glm-4.7-flash:high PASS 100% 77 in → 406 out (436 reasoning)
git submodule status
z-ai/glm-4.7-flash:medium PASS 100% 79 in → 319 out (331 reasoning)
git submodule status
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 80 in → 328 out (321 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
z-ai/glm-4.7-flash:xhigh PASS 100% 81 in → 330 out (345 reasoning)
git submodule status
z-ai/glm-5.2:high PASS 100% 86 in → 161 out (156 reasoning)
git submodule status
z-ai/glm-5.2:high__json_schema PASS 100% 266 in → 126 out (117 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
z-ai/glm-5.2:low PASS 100% 86 in → 233 out (228 reasoning)
git submodule status
z-ai/glm-5.2:medium PASS 100% 85 in → 86 out (81 reasoning)
git submodule status
z-ai/glm-5.2:none PASS 100% 80 in → 4 out (0 reasoning)
git submodule status
z-ai/glm-5.2:xhigh PASS 100% 87 in → 601 out (596 reasoning)
git submodule status
z-ai/glm-5.2:xhigh__json_schema PASS 100% 87 in → 456 out (443 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
anthropic/claude-haiku-4.5:high FAIL 0% 124 in → 8,755 out (2,827 reasoning)
``` git config --file .gitmodules -l ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 284 in → 1,000 out (598 reasoning)
git config --file .gitmodules -l
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules -l"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:low FAIL 0% 124 in → 956 out (659 reasoning)
``` git config --file .gitmodules -l ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 283 in → 782 out (567 reasoning)
git config --file .gitmodules --get-regexp path
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules --get-regexp path"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:medium FAIL 0% 126 in → 749 out (598 reasoning)
``` git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 281 in → 574 out (492 reasoning)
git config --file .gitmodules -l
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules -l"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:none FAIL 0% 95 in → 25 out (0 reasoning)
```bash git config --file .gitmodules --name-only --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 257 in → 25 out (0 reasoning)
git config --file .gitmodules --name-only --get-regexp path
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules --name-only --get-regexp path"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:xhigh FAIL 0% 121 in → 6,694 out (2,518 reasoning)
``` git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 286 in → 1,114 out (669 reasoning)
git config --file .gitmodules -l
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules -l"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.7:max FAIL 0% 121 in → 26 out (0 reasoning)
``` git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:high FAIL 0% 116 in → 19 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:low FAIL 0% 109 in → 19 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:max FAIL 0% 111 in → 19 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 324 in → 26 out (0 reasoning)
git config --file .gitmodules --get-regexp path
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules --get-regexp path"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:medium FAIL 0% 113 in → 19 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:none FAIL 0% 111 in → 23 out (0 reasoning)
``` git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-opus-4.8:xhigh FAIL 0% 116 in → 19 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-sonnet-4.6:high FAIL 0% 94 in → 323 out (169 reasoning)
`git submodule`
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 89 in → 212 out (172 reasoning)
`git submodule`
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
anthropic/claude-sonnet-5:xhigh FAIL 0% 111 in → 23 out (0 reasoning)
``` git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 79 in → 371 out (448 reasoning)
git submodule summary
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
arcee-ai/trinity-mini:high__json_schema FAIL 0% 80 in → 346 out (410 reasoning)
gitsubmodule status
JSON Schema Structured Output
(raw) { "command": "gitsubmodule status" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
arcee-ai/trinity-mini:low FAIL 0% 83 in → 324 out (370 reasoning)
gitsubmodule status
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 83 in → 472 out (549 reasoning)
git submodule summary
JSON Schema Structured Output
(raw) { "command": "git submodule summary" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 76 in → 555 out (655 reasoning)
git submodulestatus
JSON Schema Structured Output
(raw) { "command": "git submodulestatus" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 82 in → 525 out (628 reasoning)
git submodulestatus
JSON Schema Structured Output
(raw) { "command": "git submodulestatus" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
deepseek/deepseek-v4-flash-0731:none FAIL 0% 77 in → 14 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
ibm-granite/granite-4.1-8b FAIL 0% 92 in → 4 out
git submodule list
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output: .git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high FAIL 0% 246 in → 230 out (269 reasoning)
gitsubmodule status
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 81 in → 442 out (465 reasoning)
git config --get-regexp submodule\.*
JSON Schema Structured Output
(raw) {"command": "git config --get-regexp submodule\\.*"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
moonshotai/kimi-k2.7-code:low FAIL 0% 78 in → 961 out (1,096 reasoning)
git config --file .gitmodules --get-regexp '^submodule\.'
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 161 in → 613 out (587 reasoning)
git config --file .gitmodules --get-regexp '^submodule\..*\.path$'
JSON Schema Structured Output
(raw) {"command":"git config --file .gitmodules --get-regexp '^submodule\\..*\\.path$'"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k3:max FAIL 0% 157 in → 199 out (173 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 99 in → 160 out (174 reasoning)
git submodule --list
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 101 in → 354 out (374 reasoning)
git submodule --list
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-mini:none FAIL 0% 77 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-mini:none__json_schema FAIL 0% 108 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:high FAIL 0% 78 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 112 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:low FAIL 0% 80 in → 15 out (0 reasoning)
git submodule foreach --quiet 'echo $name'
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 116 in → 33 out (0 reasoning)
git config --file .gitmodules --get-regexp '^submodule\..*\.path$'
JSON Schema Structured Output
(raw) {"command":"git config --file .gitmodules --get-regexp '^submodule\\..*\\.path$'"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:medium FAIL 0% 76 in → 32 out (0 reasoning)
git config --file .gitmodules --get-regexp '^submodule\..*\.path$' | awk '{print $2}'
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 112 in → 33 out (0 reasoning)
git config --file .gitmodules --get-regexp '^submodule\..*\.path$'
JSON Schema Structured Output
(raw) {"command":"git config --file .gitmodules --get-regexp '^submodule\\..*\\.path$'"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.4-nano:none__json_schema FAIL 0% 109 in → 33 out (0 reasoning)
git config --file .gitmodules --get-regexp '^submodule\..*\.path$'
JSON Schema Structured Output
(raw) {"command":"git config --file .gitmodules --get-regexp '^submodule\\..*\\.path$'"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.6-luna:low__json_schema FAIL 0% 113 in → 193 out (167 reasoning)
git config --file .gitmodules --get-regexp path
JSON Schema Structured Output
(raw) {"command":"git config --file .gitmodules --get-regexp path"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-5.6-terra:medium FAIL 0% 73 in → 101 out (83 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-oss-120b:high__json_schema FAIL 0% 135 in → 1,714 out (1,871 reasoning)
finalgit submodule status
JSON Schema Structured Output
(raw) {"command":"finalgit submodule status"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-oss-120b:low__json_schema FAIL 0% 138 in → 99 out (85 reasoning)
finalgit submodule
JSON Schema Structured Output
(raw) {"command":"finalgit submodule"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
openai/gpt-oss-120b:medium__json_schema FAIL 0% 139 in → 320 out (336 reasoning)
final
JSON Schema Structured Output
(raw) {"command":"final"}
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-35b-a3b:medium FAIL 0% 96 in → 3,378 out (3,363 reasoning)
git submodule foreach --quiet 'echo $name'
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8f5493f4-a316-9aa2-af2e-e86599e23452","request_id":"8f5493f4-a316-9aa2-af2e-e86599e23452"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-403f7592-3d05-9cf2-8661-3d2782ad5190","request_id":"403f7592-3d05-9cf2-8661-3d2782ad5190"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c23a4270-14ad-981e-9688-c80052539309","request_id":"c23a4270-14ad-981e-9688-c80052539309"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8d40466b-595d-9ec0-9edb-33cb89f580c5","request_id":"8d40466b-595d-9ec0-9edb-33cb89f580c5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5f9cf43a-93a4-95fb-a922-a2f62792e8df","request_id":"5f9cf43a-93a4-95fb-a922-a2f62792e8df"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e1400804-d40c-9c09-a008-35ac27be4706"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7552910c-b283-99c9-b015-2007d0f02d51"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 97 in → 2,760 out (2,741 reasoning)
git config --file .gitmodules --get-regexp '^submodule'
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8124c44f-1c51-9501-a78b-6859292b8d50"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2d510271-91d0-974e-b7ea-f06e00e9df6b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-075b6c74-0e45-9539-8771-537b0dda9510"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4fb75b1c-7e9a-90c9-a7f0-8aa81a88f0f0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fc01721c-6f35-926b-9b2d-8ac5474b016a","request_id":"fc01721c-6f35-926b-9b2d-8ac5474b016a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14fe33f1-1136-9d1b-b171-bc9f465c696d","request_id":"14fe33f1-1136-9d1b-b171-bc9f465c696d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2b0e392c-6129-9d7f-a33d-bd4d1e2f9333","request_id":"2b0e392c-6129-9d7f-a33d-bd4d1e2f9333"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-57c62ed3-3f84-9124-ae32-e88ab2125529","request_id":"57c62ed3-3f84-9124-ae32-e88ab2125529"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-830ebfca-06cc-96e5-9a49-ccd479c0dc06","request_id":"830ebfca-06cc-96e5-9a49-ccd479c0dc06"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6f3471c5-de0c-9be6-9634-ad27b7b3e7af","request_id":"6f3471c5-de0c-9be6-9634-ad27b7b3e7af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-571c7e14-cb49-9fc1-9ce7-dba0064ce848","request_id":"571c7e14-cb49-9fc1-9ce7-dba0064ce848"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-65b94d22-108d-98d6-8ba7-a20ef31fc94e","request_id":"65b94d22-108d-98d6-8ba7-a20ef31fc94e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7d48d9a3-5704-938e-900f-842fd7ddbf47","request_id":"7d48d9a3-5704-938e-900f-842fd7ddbf47"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6f50d0bc-c84f-95b5-8fa4-0a4e569a0ed1","request_id":"6f50d0bc-c84f-95b5-8fa4-0a4e569a0ed1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 86 in → 7 out (0 reasoning)
git submodule status<|end_message|>
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output: git submodule{"command": "git submodule"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 85 in → 6 out (0 reasoning)
git submodule<|end_message|>
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
x-ai/grok-4.3:max FAIL 0% 257 in → 753 out (737 reasoning)
```sh git config --file .gitmodules --get-regexp path ```
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
x-ai/grok-4.3:xhigh FAIL 0% 259 in → 1,165 out (1,154 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
z-ai/glm-4.7-flash:high__json_schema FAIL 0% 81 in → 684 out (701 reasoning)
git config --file .gitmodules --get-regexp path
JSON Schema Structured Output
(raw) { "command": "git config --file .gitmodules --get-regexp path" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
z-ai/glm-4.7-flash:low FAIL 0% 82 in → 857 out (931 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 78 in → 12 out (0 reasoning)
git config --file .gitmodules --get-regexp path
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
z-ai/glm-4.7-flash:none__json_schema FAIL 0% 79 in → 29 out (0 reasoning)
git config --file .gitmodules --get-regexp path | cut -d' ' -f2
JSON Schema Structured Output
(raw) { "command": "git config --file .gitmodules --get-regexp path | cut -d' ' -f2" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none__json_schema FAIL 0% 81 in → 17 out (0 reasoning)
git config -f .gitmodules --list
JSON Schema Structured Output
(raw) { "command": "git config -f .gitmodules --list" }
Failure: Command did not match accepted alternatives: ['git submodule', 'git submodule status']