Check submodule status
Tests ability to check submodule status for modifications. Evaluates submodule-status inspection.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
Prompt
Run the command to show the status of all submodules. Output ONLY the git command, nothing else.
Expected
git submodule status

Scoped model quality, cost, API time, and token usage for submodule_usage/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 114 in → 17 out (19 reasoning)
git submodule status
anthropic/claude-fable-5:high__json_schema PASS 100% 324 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:low PASS 100% 117 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-fable-5:low__json_schema PASS 100% 322 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:max PASS 100% 115 in → 49 out (18 reasoning)
git submodule status
anthropic/claude-fable-5:max__json_schema PASS 100% 322 in → 65 out (24 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:medium PASS 100% 113 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-fable-5:medium__json_schema PASS 100% 325 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-fable-5:xhigh PASS 100% 115 in → 20 out (27 reasoning)
git submodule status
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 322 in → 27 out (27 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 291 in → 268 out (241 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-haiku-4.5:low PASS 100% 123 in → 106 out (102 reasoning)
git submodule status
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 292 in → 216 out (196 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 290 in → 131 out (117 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 288 in → 127 out (113 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-opus-4.6:high__json_schema PASS 100% 259 in → 41 out (13 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.6:low__json_schema PASS 100% 261 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 257 in → 54 out (26 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 262 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.6:none__json_schema PASS 100% 260 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 261 in → 41 out (13 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:high__json_schema PASS 100% 328 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:low PASS 100% 117 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:low__json_schema PASS 100% 329 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:max__json_schema PASS 100% 329 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:medium PASS 100% 118 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 329 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:none PASS 100% 119 in → 9 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.7:none__json_schema PASS 100% 333 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 321 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:high PASS 100% 113 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.8:high__json_schema PASS 100% 324 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:low PASS 100% 113 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.8:low__json_schema PASS 100% 323 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:max PASS 100% 110 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.8:max__json_schema PASS 100% 321 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:medium PASS 100% 117 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 323 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 326 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-opus-4.8:xhigh PASS 100% 113 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 318 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 259 in → 45 out (19 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 258 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 259 in → 143 out (115 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 256 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 261 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-5:high__json_schema PASS 100% 323 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:low PASS 100% 114 in → 7 out (0 reasoning)
git submodule status
anthropic/claude-sonnet-5:low__json_schema PASS 100% 324 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 328 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 319 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 320 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
arcee-ai/trinity-large-thinking:high PASS 100% 79 in → 198 out (192 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:low PASS 100% 82 in → 172 out (166 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:medium PASS 100% 86 in → 256 out (250 reasoning)
git submodule status
arcee-ai/trinity-large-thinking:xhigh PASS 100% 81 in → 332 out (328 reasoning)
git submodule status
arcee-ai/trinity-mini:high PASS 100% 82 in → 385 out (433 reasoning)
git submodule status
arcee-ai/trinity-mini:high__json_schema PASS 100% 87 in → 393 out (459 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
arcee-ai/trinity-mini:low PASS 100% 87 in → 239 out (281 reasoning)
git submodule status
arcee-ai/trinity-mini:low__json_schema PASS 100% 83 in → 292 out (343 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 81 in → 338 out (386 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command":"git submodule status" }
arcee-ai/trinity-mini:xhigh PASS 100% 80 in → 280 out (347 reasoning)
git submodule status
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 82 in → 238 out (280 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command":"git submodule status" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 158 in → 31 out (28 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 78 in → 62 out (55 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 84 in → 174 out (182 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 84 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 169 in → 97 out (102 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 171 in → 84 out (87 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 82 in → 50 out (48 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 85 in → 124 out (121 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 82 in → 4 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 78 in → 13 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 161 in → 57 out (58 reasoning)
git submodule status
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 80 in → 97 out (91 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:high PASS 100% 84 in → 78 out (72 reasoning)
git submodule status
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 83 in → 105 out (91 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:low PASS 100% 81 in → 54 out (49 reasoning)
git submodule status
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 77 in → 143 out (121 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:medium PASS 100% 81 in → 49 out (43 reasoning)
git submodule status
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 81 in → 105 out (91 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:none PASS 100% 80 in → 5 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 85 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 78 in → 87 out (93 reasoning)
git submodule status
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 158 in → 375 out (423 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-pro:high PASS 100% 81 in → 88 out (82 reasoning)
git submodule status
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 77 in → 142 out (149 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-pro:low PASS 100% 84 in → 88 out (82 reasoning)
git submodule status
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 82 in → 313 out (300 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
deepseek/deepseek-v4-pro:medium PASS 100% 82 in → 5 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 267 in → 106 out (94 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
deepseek/deepseek-v4-pro:none PASS 100% 80 in → 5 out (0 reasoning)
git submodule status
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 265 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 160 in → 348 out (343 reasoning)
git submodule status
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 345 in → 318 out (307 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3-flash-preview:high PASS 100% 95 in → 77 out (74 reasoning)
git submodule status
google/gemini-3-flash-preview:high__json_schema PASS 100% 154 in → 407 out (400 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3-flash-preview:low PASS 100% 95 in → 76 out (73 reasoning)
git submodule status
google/gemini-3-flash-preview:low__json_schema PASS 100% 153 in → 54 out (47 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3-flash-preview:medium PASS 100% 94 in → 175 out (172 reasoning)
git submodule status
google/gemini-3-flash-preview:medium__json_schema PASS 100% 97 in → 217 out (210 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3-flash-preview:xhigh PASS 100% 97 in → 97 out (94 reasoning)
git submodule status
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 150 in → 208 out (200 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 95 in → 191 out (188 reasoning)
git submodule status
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 153 in → 158 out (150 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 94 in → 115 out (112 reasoning)
git submodule status
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 95 in → 116 out (104 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 97 in → 100 out (97 reasoning)
git submodule status
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 95 in → 121 out (108 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 95 in → 299 out (296 reasoning)
git submodule status
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 94 in → 157 out (144 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.1-pro-preview:high PASS 100% 95 in → 314 out (311 reasoning)
git submodule status
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 150 in → 246 out (239 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.1-pro-preview:low PASS 100% 95 in → 129 out (126 reasoning)
git submodule status
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 151 in → 49 out (41 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-pro-preview:medium PASS 100% 94 in → 209 out (206 reasoning)
git submodule status
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 148 in → 233 out (225 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 96 in → 191 out (188 reasoning)
git submodule status
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 152 in → 310 out (303 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash-lite:high PASS 100% 93 in → 222 out (219 reasoning)
git submodule status
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 152 in → 355 out (348 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash-lite:low PASS 100% 96 in → 3 out (0 reasoning)
git submodule status
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 94 in → 8 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash-lite:max PASS 100% 97 in → 175 out (172 reasoning)
git submodule status
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 96 in → 268 out (260 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash-lite:medium PASS 100% 93 in → 102 out (99 reasoning)
git submodule status
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 146 in → 190 out (183 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 96 in → 179 out (176 reasoning)
git submodule status
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 95 in → 188 out (181 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash:high PASS 100% 93 in → 333 out (330 reasoning)
git submodule status
google/gemini-3.5-flash:high__json_schema PASS 100% 150 in → 183 out (175 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.5-flash:low PASS 100% 96 in → 121 out (118 reasoning)
git submodule status
google/gemini-3.5-flash:low__json_schema PASS 100% 151 in → 38 out (25 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.5-flash:medium PASS 100% 94 in → 250 out (247 reasoning)
git submodule status
google/gemini-3.5-flash:medium__json_schema PASS 100% 148 in → 295 out (288 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.5-flash:xhigh PASS 100% 97 in → 194 out (191 reasoning)
git submodule status
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 154 in → 282 out (274 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
google/gemini-3.6-flash:high PASS 100% 94 in → 163 out (160 reasoning)
git submodule status
google/gemini-3.6-flash:high__json_schema PASS 100% 151 in → 213 out (206 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:low PASS 100% 94 in → 55 out (52 reasoning)
git submodule status
google/gemini-3.6-flash:low__json_schema PASS 100% 95 in → 7 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:max PASS 100% 91 in → 249 out (246 reasoning)
git submodule status
google/gemini-3.6-flash:max__json_schema PASS 100% 96 in → 160 out (153 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemini-3.6-flash:medium PASS 100% 97 in → 143 out (140 reasoning)
git submodule status
google/gemini-3.6-flash:medium__json_schema PASS 100% 98 in → 176 out (163 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemini-3.6-flash:xhigh PASS 100% 92 in → 209 out (206 reasoning)
git submodule status
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 90 in → 325 out (318 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
google/gemma-4-26b-a4b-it:high PASS 100% 112 in → 174 out (170 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 108 in → 173 out (162 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:low PASS 100% 112 in → 208 out (224 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 111 in → 247 out (239 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:medium PASS 100% 110 in → 137 out (137 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 114 in → 187 out (191 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:none PASS 100% 107 in → 4 out (0 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 107 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 113 in → 144 out (147 reasoning)
git submodule status
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 106 in → 182 out (173 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:high PASS 100% 110 in → 137 out (137 reasoning)
git submodule status
google/gemma-4-31b-it:high__json_schema PASS 100% 108 in → 153 out (141 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:low PASS 100% 110 in → 121 out (117 reasoning)
git submodule status
google/gemma-4-31b-it:medium PASS 100% 110 in → 140 out (133 reasoning)
git submodule status
google/gemma-4-31b-it:medium__json_schema PASS 100% 106 in → 117 out (109 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:none PASS 100% 111 in → 4 out (0 reasoning)
git submodule status
google/gemma-4-31b-it:none__json_schema PASS 100% 107 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
google/gemma-4-31b-it:xhigh PASS 100% 111 in → 175 out (145 reasoning)
git submodule status
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 108 in → 107 out (91 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
inclusionai/ling-2.6-flash__json_schema PASS 100% 113 in → 15 out
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
liquid/lfm-2-24b-a2b PASS 100% 92 in → 5 out
git submodule status
minimax/minimax-m2.5:high PASS 100% 115 in → 83 out (99 reasoning)
git submodule status
minimax/minimax-m2.5:high__json_schema PASS 100% 121 in → 1,362 out (1,350 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:low PASS 100% 114 in → 729 out (819 reasoning)
git submodule status
minimax/minimax-m2.5:low__json_schema PASS 100% 123 in → 73 out (61 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:medium PASS 100% 127 in → 92 out (87 reasoning)
git submodule status
minimax/minimax-m2.5:medium__json_schema PASS 100% 110 in → 242 out (231 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.5:xhigh PASS 100% 114 in → 147 out (143 reasoning)
git submodule status
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 123 in → 118 out (106 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.7:high PASS 100% 114 in → 207 out (201 reasoning)
git submodule status
minimax/minimax-m2.7:high__json_schema PASS 100% 241 in → 246 out (237 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
minimax/minimax-m2.7:low PASS 100% 119 in → 165 out (161 reasoning)
git submodule status
minimax/minimax-m2.7:low__json_schema PASS 100% 243 in → 422 out (413 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
minimax/minimax-m2.7:medium PASS 100% 112 in → 190 out (185 reasoning)
git submodule status
minimax/minimax-m2.7:medium__json_schema PASS 100% 241 in → 153 out (140 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m2.7:xhigh PASS 100% 113 in → 542 out (536 reasoning)
git submodule status
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 108 in → 182 out (187 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:high PASS 100% 380 in → 36 out (36 reasoning)
git submodule status
minimax/minimax-m3:high__json_schema PASS 100% 235 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
minimax/minimax-m3:low PASS 100% 247 in → 20 out (19 reasoning)
git submodule status
minimax/minimax-m3:low__json_schema PASS 100% 199 in → 9 out (8 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
minimax/minimax-m3:medium PASS 100% 195 in → 34 out (28 reasoning)
git submodule status
minimax/minimax-m3:medium__json_schema PASS 100% 236 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
minimax/minimax-m3:xhigh PASS 100% 246 in → 32 out (34 reasoning)
git submodule status
minimax/minimax-m3:xhigh__json_schema PASS 100% 233 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
mistralai/devstral-2512__json_schema PASS 100% 92 in → 10 out
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:high PASS 100% 101 in → 173 out (192 reasoning)
git submodule status
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 90 in → 187 out (195 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:low PASS 100% 101 in → 127 out (140 reasoning)
git submodule status
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 91 in → 173 out (191 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:medium PASS 100% 102 in → 357 out (357 reasoning)
git submodule status
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 89 in → 210 out (228 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 89 in → 10 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 103 in → 113 out (123 reasoning)
git submodule status
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 87 in → 274 out (315 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
moonshotai/kimi-k2.6:high PASS 100% 79 in → 266 out (261 reasoning)
git submodule status
moonshotai/kimi-k2.6:low PASS 100% 76 in → 115 out (110 reasoning)
git submodule status
moonshotai/kimi-k2.6:low__json_schema PASS 100% 80 in → 280 out (119 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k2.6:medium PASS 100% 83 in → 170 out (185 reasoning)
git submodule status
moonshotai/kimi-k2.6:xhigh PASS 100% 78 in → 124 out (157 reasoning)
git submodule status
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 77 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k2.7-code:high PASS 100% 80 in → 40 out (35 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:low PASS 100% 80 in → 88 out (83 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 165 in → 68 out (59 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
moonshotai/kimi-k2.7-code:medium PASS 100% 81 in → 131 out (126 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 79 in → 193 out (183 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 81 in → 92 out (87 reasoning)
git submodule status
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 78 in → 36 out (24 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
moonshotai/kimi-k3:high PASS 100% 156 in → 51 out (33 reasoning)
git submodule status
moonshotai/kimi-k3:high__json_schema PASS 100% 157 in → 78 out (60 reasoning)
git submodule status
moonshotai/kimi-k3:low PASS 100% 157 in → 102 out (84 reasoning)
git submodule status
moonshotai/kimi-k3:low__json_schema PASS 100% 152 in → 49 out (31 reasoning)
git submodule status
moonshotai/kimi-k3:max PASS 100% 161 in → 55 out (37 reasoning)
git submodule status
moonshotai/kimi-k3:max__json_schema PASS 100% 157 in → 57 out (39 reasoning)
git submodule status
moonshotai/kimi-k3:medium PASS 100% 158 in → 44 out (26 reasoning)
git submodule status
moonshotai/kimi-k3:medium__json_schema PASS 100% 158 in → 46 out (28 reasoning)
git submodule status
moonshotai/kimi-k3:xhigh PASS 100% 161 in → 49 out (31 reasoning)
git submodule status
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 157 in → 46 out (28 reasoning)
git submodule status
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 104 in → 193 out (207 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status" }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 101 in → 265 out (295 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 102 in → 144 out (156 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 100 in → 71 out (63 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 105 in → 49 out (34 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 103 in → 120 out (135 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 105 in → 103 out (86 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 103 in → 38 out (36 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 106 in → 130 out (112 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 103 in → 5 out (0 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 104 in → 10 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 100 in → 324 out (292 reasoning)
git submodule status
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 107 in → 73 out (57 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
openai/gpt-5.4-mini:high PASS 100% 78 in → 136 out (126 reasoning)
git submodule status
openai/gpt-5.4-mini:xhigh PASS 100% 79 in → 647 out (637 reasoning)
git submodule status
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 116 in → 351 out (333 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.4-nano:low PASS 100% 78 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.4-nano:xhigh PASS 100% 77 in → 56 out (46 reasoning)
git submodule status
openai/gpt-5.4:xhigh PASS 100% 83 in → 495 out (485 reasoning)
git submodule status
openai/gpt-5.4:xhigh__json_schema PASS 100% 112 in → 2,368 out (2,350 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:high PASS 100% 76 in → 88 out (78 reasoning)
git submodule status
openai/gpt-5.5:high__json_schema PASS 100% 114 in → 332 out (314 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:low__json_schema PASS 100% 113 in → 17 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
openai/gpt-5.5:medium PASS 100% 80 in → 73 out (63 reasoning)
git submodule status
openai/gpt-5.5:medium__json_schema PASS 100% 111 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:none__json_schema PASS 100% 111 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.5:xhigh PASS 100% 75 in → 372 out (362 reasoning)
git submodule status
openai/gpt-5.5:xhigh__json_schema PASS 100% 108 in → 535 out (516 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
openai/gpt-5.6-luna:low__json_schema PASS 100% 113 in → 84 out (66 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:max PASS 100% 79 in → 149 out (139 reasoning)
git submodule status
openai/gpt-5.6-luna:medium__json_schema PASS 100% 114 in → 60 out (42 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:none PASS 100% 77 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-luna:none__json_schema PASS 100% 113 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-luna:xhigh PASS 100% 80 in → 120 out (110 reasoning)
git submodule status
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 115 in → 66 out (48 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:high PASS 100% 79 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:high__json_schema PASS 100% 112 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:low PASS 100% 78 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:low__json_schema PASS 100% 113 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:max PASS 100% 75 in → 95 out (85 reasoning)
git submodule status
openai/gpt-5.6-sol:max__json_schema PASS 100% 119 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:medium PASS 100% 82 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:medium__json_schema PASS 100% 113 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:none PASS 100% 77 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:none__json_schema PASS 100% 115 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-sol:xhigh PASS 100% 83 in → 8 out (0 reasoning)
git submodule status
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 115 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:low PASS 100% 79 in → 38 out (28 reasoning)
git submodule status
openai/gpt-5.6-terra:max PASS 100% 75 in → 96 out (86 reasoning)
git submodule status
openai/gpt-5.6-terra:max__json_schema PASS 100% 114 in → 520 out (502 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:none__json_schema PASS 100% 116 in → 16 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-5.6-terra:xhigh PASS 100% 80 in → 8 out (0 reasoning)
git submodule status
openai/gpt-oss-120b:high PASS 100% 137 in → 363 out (413 reasoning)
git submodule status
openai/gpt-oss-120b:high__json_schema PASS 100% 140 in → 293 out (312 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-120b:low PASS 100% 151 in → 58 out (0 reasoning)
git submodule status
openai/gpt-oss-120b:low__json_schema PASS 100% 135 in → 59 out (46 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-120b:medium PASS 100% 140 in → 106 out (104 reasoning)
git submodule status
openai/gpt-oss-120b:medium__json_schema PASS 100% 143 in → 152 out (129 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-120b:xhigh PASS 100% 126 in → 344 out (385 reasoning)
git submodule status
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 124 in → 281 out (301 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-20b:high PASS 100% 139 in → 470 out (438 reasoning)
git submodule status
openai/gpt-oss-20b:high__json_schema PASS 100% 141 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status" }
openai/gpt-oss-20b:low PASS 100% 126 in → 55 out (46 reasoning)
git submodule status
openai/gpt-oss-20b:low__json_schema PASS 100% 136 in → 25 out (9 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
openai/gpt-oss-20b:medium PASS 100% 136 in → 109 out (104 reasoning)
git submodule status
openai/gpt-oss-20b:medium__json_schema PASS 100% 143 in → 100 out (96 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
openai/gpt-oss-20b:xhigh PASS 100% 139 in → 216 out (196 reasoning)
git submodule status
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 142 in → 316 out (337 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
poolside/laguna-m.1:high PASS 100% 106 in → 232 out (226 reasoning)
git submodule status
poolside/laguna-m.1:high__json_schema PASS 100% 101 in → 176 out (166 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-m.1:low PASS 100% 98 in → 232 out (226 reasoning)
git submodule status
poolside/laguna-m.1:low__json_schema PASS 100% 103 in → 181 out (167 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-m.1:medium PASS 100% 104 in → 238 out (232 reasoning)
git submodule status
poolside/laguna-m.1:medium__json_schema PASS 100% 99 in → 139 out (124 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-m.1:none__json_schema PASS 100% 105 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-m.1:xhigh PASS 100% 105 in → 220 out (214 reasoning)
git submodule status
poolside/laguna-m.1:xhigh__json_schema PASS 100% 103 in → 164 out (149 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:high PASS 100% 102 in → 284 out (280 reasoning)
git submodule status
poolside/laguna-xs-2.1:high__json_schema PASS 100% 103 in → 226 out (211 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:low PASS 100% 101 in → 336 out (332 reasoning)
git submodule status
poolside/laguna-xs-2.1:low__json_schema PASS 100% 100 in → 364 out (354 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs-2.1:medium PASS 100% 101 in → 242 out (238 reasoning)
git submodule status
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 103 in → 580 out (565 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:none PASS 100% 104 in → 3 out (0 reasoning)
git submodule status
poolside/laguna-xs-2.1:none__json_schema PASS 100% 99 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs-2.1:xhigh PASS 100% 103 in → 318 out (314 reasoning)
git submodule status
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 104 in → 198 out (183 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs.2:high PASS 100% 138 in → 128 out (122 reasoning)
git submodule status
poolside/laguna-xs.2:high__json_schema PASS 100% 135 in → 196 out (186 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:low PASS 100% 139 in → 56 out (50 reasoning)
git submodule status
poolside/laguna-xs.2:low__json_schema PASS 100% 140 in → 217 out (207 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:medium PASS 100% 140 in → 247 out (241 reasoning)
git submodule status
poolside/laguna-xs.2:medium__json_schema PASS 100% 137 in → 107 out (92 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
poolside/laguna-xs.2:none PASS 100% 144 in → 5 out (0 reasoning)
git submodule status
poolside/laguna-xs.2:none__json_schema PASS 100% 141 in → 9 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 137 in → 307 out (297 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
qwen/qwen3.6-27b:high PASS 100% 101 in → 616 out (677 reasoning)
git submodule status
qwen/qwen3.6-27b:high__json_schema PASS 100% 98 in → 702 out (732 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:low PASS 100% 100 in → 742 out (782 reasoning)
git submodule status
qwen/qwen3.6-27b:low__json_schema PASS 100% 101 in → 1,034 out (1 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:medium PASS 100% 100 in → 167 out (193 reasoning)
git submodule status
qwen/qwen3.6-27b:medium__json_schema PASS 100% 101 in → 204 out (226 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-27b:none PASS 100% 102 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.6-27b:none__json_schema PASS 100% 100 in → 8 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
qwen/qwen3.6-27b:xhigh PASS 100% 100 in → 375 out (367 reasoning)
git submodule status
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 99 in → 324 out (336 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:high PASS 100% 99 in → 332 out (359 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 98 in → 1,046 out (1,138 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:low PASS 100% 100 in → 628 out (620 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 101 in → 865 out (926 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 99 in → 511 out (503 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 102 in → 269 out (279 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:none PASS 100% 100 in → 4 out (0 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 101 in → 14 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 100 in → 472 out (464 reasoning)
git submodule status
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 103 in → 287 out (298 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
qwen/qwen3.6-flash:high PASS 100% 96 in → 284 out (276 reasoning)
git submodule status
qwen/qwen3.6-flash:low PASS 100% 100 in → 815 out (807 reasoning)
git submodule status
qwen/qwen3.6-flash:medium PASS 100% 99 in → 331 out (323 reasoning)
git submodule status
qwen/qwen3.6-flash:none PASS 100% 101 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.6-flash:xhigh PASS 100% 99 in → 984 out (976 reasoning)
git submodule status
qwen/qwen3.7-flash:high PASS 100% 99 in → 321 out (313 reasoning)
git submodule status
qwen/qwen3.7-flash:low PASS 100% 99 in → 325 out (317 reasoning)
git submodule status
qwen/qwen3.7-flash:max PASS 100% 100 in → 791 out (783 reasoning)
git submodule status
qwen/qwen3.7-flash:medium PASS 100% 98 in → 295 out (287 reasoning)
git submodule status
qwen/qwen3.7-flash:none PASS 100% 101 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-flash:none__json_schema PASS 100% 102 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-flash:xhigh PASS 100% 98 in → 676 out (668 reasoning)
git submodule status
qwen/qwen3.7-max:high PASS 100% 102 in → 110 out (102 reasoning)
git submodule status
qwen/qwen3.7-max:low PASS 100% 103 in → 44 out (36 reasoning)
git submodule status
qwen/qwen3.7-max:medium PASS 100% 99 in → 106 out (98 reasoning)
git submodule status
qwen/qwen3.7-max:none PASS 100% 101 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-max:xhigh PASS 100% 101 in → 447 out (439 reasoning)
git submodule status
qwen/qwen3.7-plus:high PASS 100% 98 in → 417 out (409 reasoning)
git submodule status
qwen/qwen3.7-plus:low PASS 100% 97 in → 424 out (416 reasoning)
git submodule status
qwen/qwen3.7-plus:medium PASS 100% 101 in → 193 out (185 reasoning)
git submodule status
qwen/qwen3.7-plus:none PASS 100% 99 in → 3 out (0 reasoning)
git submodule status
qwen/qwen3.7-plus:xhigh PASS 100% 99 in → 190 out (182 reasoning)
git submodule status
tencent/hy3:high PASS 100% 91 in → 307 out (345 reasoning)
git submodule status
tencent/hy3:high__json_schema PASS 100% 89 in → 565 out (611 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:low PASS 100% 84 in → 317 out (312 reasoning)
git submodule status
tencent/hy3:low__json_schema PASS 100% 91 in → 178 out (196 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:medium PASS 100% 93 in → 393 out (388 reasoning)
git submodule status
tencent/hy3:medium__json_schema PASS 100% 86 in → 398 out (428 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:none__json_schema PASS 100% 91 in → 12 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
tencent/hy3:xhigh PASS 100% 88 in → 356 out (351 reasoning)
git submodule status
tencent/hy3:xhigh__json_schema PASS 100% 86 in → 447 out (489 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:high PASS 100% 88 in → 149 out (139 reasoning)
git submodule status
thinkingmachines/inkling-small:high__json_schema PASS 100% 84 in → 125 out (108 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:low PASS 100% 87 in → 31 out (24 reasoning)
git submodule status
thinkingmachines/inkling-small:low__json_schema PASS 100% 90 in → 53 out (37 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:max PASS 100% 86 in → 67 out (57 reasoning)
git submodule status
thinkingmachines/inkling-small:max__json_schema PASS 100% 86 in → 184 out (167 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:medium__json_schema PASS 100% 93 in → 117 out (100 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling-small:xhigh PASS 100% 86 in → 174 out (163 reasoning)
git submodule status
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 88 in → 120 out (106 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
thinkingmachines/inkling:high PASS 100% 88 in → 127 out (116 reasoning)
git submodule status
thinkingmachines/inkling:low PASS 100% 87 in → 23 out (12 reasoning)
git submodule status
thinkingmachines/inkling:max PASS 100% 85 in → 63 out (61 reasoning)
git submodule status
thinkingmachines/inkling:max__json_schema PASS 100% 86 in → 349 out (332 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
thinkingmachines/inkling:medium PASS 100% 89 in → 98 out (88 reasoning)
git submodule status
thinkingmachines/inkling:xhigh PASS 100% 84 in → 44 out (34 reasoning)
git submodule status
thinkingmachines/inkling:xhigh__json_schema PASS 100% 90 in → 105 out (89 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
x-ai/grok-4.3:high PASS 100% 256 in → 118 out (115 reasoning)
git submodule status
x-ai/grok-4.3:high__json_schema PASS 100% 309 in → 319 out (312 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:low PASS 100% 262 in → 82 out (79 reasoning)
git submodule status
x-ai/grok-4.3:low__json_schema PASS 100% 319 in → 263 out (256 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:max PASS 100% 257 in → 78 out (75 reasoning)
git submodule status
x-ai/grok-4.3:max__json_schema PASS 100% 317 in → 259 out (252 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:medium PASS 100% 264 in → 329 out (326 reasoning)
git submodule status
x-ai/grok-4.3:medium__json_schema PASS 100% 322 in → 293 out (282 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
x-ai/grok-4.3:none PASS 100% 259 in → 3 out (0 reasoning)
git submodule status
x-ai/grok-4.3:none__json_schema PASS 100% 309 in → 7 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.3:xhigh PASS 100% 258 in → 137 out (134 reasoning)
git submodule status
x-ai/grok-4.3:xhigh__json_schema PASS 100% 317 in → 275 out (268 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.5:high PASS 100% 285 in → 147 out (144 reasoning)
git submodule status
x-ai/grok-4.5:high__json_schema PASS 100% 352 in → 138 out (130 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command":"git submodule status"}
x-ai/grok-4.5:low__json_schema PASS 100% 351 in → 143 out (135 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:max PASS 100% 282 in → 133 out (130 reasoning)
git submodule status
x-ai/grok-4.5:max__json_schema PASS 100% 349 in → 281 out (273 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:medium__json_schema PASS 100% 346 in → 167 out (159 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
x-ai/grok-4.5:xhigh PASS 100% 278 in → 175 out (172 reasoning)
git submodule status
x-ai/grok-4.5:xhigh__json_schema PASS 100% 353 in → 253 out (245 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
z-ai/glm-4.7-flash:high PASS 100% 81 in → 419 out (435 reasoning)
git submodule status
z-ai/glm-4.7-flash:high__json_schema PASS 100% 82 in → 256 out (269 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
z-ai/glm-4.7-flash:low PASS 100% 78 in → 362 out (355 reasoning)
git submodule status
z-ai/glm-4.7-flash:low__json_schema PASS 100% 82 in → 504 out (528 reasoning)
git submodule status
JSON Schema Structured Output
(raw) {"command": "git submodule status"}
z-ai/glm-4.7-flash:medium PASS 100% 84 in → 432 out (467 reasoning)
git submodule status
z-ai/glm-4.7-flash:none PASS 100% 80 in → 4 out (0 reasoning)
git submodule status
z-ai/glm-4.7-flash:none__json_schema PASS 100% 81 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-4.7-flash:xhigh PASS 100% 84 in → 422 out (425 reasoning)
git submodule status
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 85 in → 550 out (594 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-5.2:high PASS 100% 89 in → 240 out (235 reasoning)
git submodule status
z-ai/glm-5.2:high__json_schema PASS 100% 94 in → 38 out (27 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-5.2:low PASS 100% 89 in → 140 out (147 reasoning)
git submodule status
z-ai/glm-5.2:low__json_schema PASS 100% 87 in → 44 out (31 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-5.2:medium PASS 100% 89 in → 163 out (0 reasoning)
git submodule status
z-ai/glm-5.2:medium__json_schema PASS 100% 89 in → 215 out (223 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-5.2:none PASS 100% 83 in → 4 out (0 reasoning)
git submodule status
z-ai/glm-5.2:none__json_schema PASS 100% 81 in → 11 out (0 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
z-ai/glm-5.2:xhigh PASS 100% 89 in → 192 out (202 reasoning)
git submodule status
z-ai/glm-5.2:xhigh__json_schema PASS 100% 91 in → 220 out (208 reasoning)
git submodule status
JSON Schema Structured Output
(raw) { "command": "git submodule status" }
anthropic/claude-haiku-4.5:high FAIL 0% 129 in → 126 out (113 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-haiku-4.5:medium FAIL 0% 124 in → 154 out (146 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-haiku-4.5:none FAIL 0% 101 in → 41 out (0 reasoning)
```bash git config --file .gitmodules --name-only --get-regexp path ``` or more directly: ```bash git submodule status ```
Failure: Expected 'git submodule status', got '```bash git config --file .gitmodules --name-only --get-regexp path ``` or more directly: ```bash git submodule status ```'
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 258 in → 25 out (0 reasoning)
git config --file .gitmodules --name-only --get-regexp path
JSON Schema Structured Output
(raw) {"command": "git config --file .gitmodules --name-only --get-regexp path"}
Failure: Expected 'git submodule status', got 'git config --file .gitmodules --name-only --get-regexp path'
anthropic/claude-haiku-4.5:xhigh FAIL 0% 125 in → 130 out (128 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:high FAIL 0% 100 in → 13 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:low FAIL 0% 96 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:max FAIL 0% 96 in → 13 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:medium FAIL 0% 96 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:none FAIL 0% 97 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.6:xhigh FAIL 0% 95 in → 13 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.7:high FAIL 0% 113 in → 11 out (0 reasoning)
`git submodule status`
Failure: Expected 'git submodule status', got '`git submodule status`'
anthropic/claude-opus-4.7:max FAIL 0% 115 in → 14 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-opus-4.7:xhigh FAIL 0% 116 in → 11 out (0 reasoning)
`git submodule status`
Failure: Expected 'git submodule status', got '`git submodule status`'
anthropic/claude-opus-4.8:none FAIL 0% 112 in → 11 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:high FAIL 0% 96 in → 45 out (19 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:low FAIL 0% 94 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:max FAIL 0% 99 in → 116 out (95 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:medium FAIL 0% 97 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:none FAIL 0% 98 in → 12 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 96 in → 44 out (18 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 260 in → 47 out (19 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
anthropic/claude-sonnet-5:high FAIL 0% 108 in → 11 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-5:medium FAIL 0% 111 in → 11 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-5:none FAIL 0% 112 in → 11 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
anthropic/claude-sonnet-5:xhigh FAIL 0% 116 in → 11 out (0 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:medium FAIL 0% 79 in → 273 out (312 reasoning)
git submodulestatus
Failure: Expected 'git submodule status', got ' git submodulestatus'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 93 in → 9 out
```bash git submodule status ```
Failure: Expected 'git submodule status', got '```bash git submodule status ```'
inclusionai/ling-2.6-flash FAIL 0% 113 in → 9 out
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
Invalid JSON. Output: .git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/devstral-2512 FAIL 0% 86 in → 10 out
```sh git submodule status ```
Failure: Expected 'git submodule status', got '```sh git submodule status ```'
mistralai/mistral-medium-3-5:none FAIL 0% 103 in → 10 out (0 reasoning)
```git git submodule status ```
Failure: Expected 'git submodule status', got '```git git submodule status ```'
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 84 in → 288 out (221 reasoning)
.gitmodules[submodule "lib"] path = lib url = ../lib-bare Submodule status: 5aa8d0a7f1efdca74d1478346ec3ff86a72a8e51 lib (heads/main)
JSON Schema Structured Output
(raw) {"command":".gitmodules[submodule \"lib\"]\n\tpath = lib\n\turl = ../lib-bare\n\nSubmodule status:\n 5aa8d0a7f1efdca74d1478346ec3ff86a72a8e51 lib (heads/main)\n"}
Failure: Expected 'git submodule status', got '.gitmodules[submodule "lib"] path = lib url = ../lib-bare Submodule status: 5aa8d0a7f1efdca74d1478346ec3ff86a72a8e51 lib (heads/main) '
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0% 167 in → 193 out (181 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command": "git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 104 in → 32,768 out (24 reasoning)
(empty output)
Failure: Expected 'git submodule status', got ''
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 101 in → 311 out (325 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 105 in → 218 out (231 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 104 in → 193 out (199 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 102 in → 8 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 104 in → 20 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 100 in → 160 out (168 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:high__json_schema FAIL 0% 112 in → 162 out (142 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:low FAIL 0% 80 in → 52 out (40 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:low__json_schema FAIL 0% 115 in → 62 out (42 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:medium FAIL 0% 79 in → 77 out (65 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:medium__json_schema FAIL 0% 113 in → 73 out (53 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:none FAIL 0% 82 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-mini:none__json_schema FAIL 0% 116 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:high FAIL 0% 80 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:high__json_schema FAIL 0% 112 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:low__json_schema FAIL 0% 115 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:medium FAIL 0% 78 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 117 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:none FAIL 0% 81 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:none__json_schema FAIL 0% 116 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4-nano:xhigh__json_schema FAIL 0% 114 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:high FAIL 0% 85 in → 146 out (134 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:high__json_schema FAIL 0% 113 in → 536 out (516 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:low FAIL 0% 77 in → 65 out (53 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:low__json_schema FAIL 0% 112 in → 57 out (37 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:medium FAIL 0% 81 in → 110 out (98 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:medium__json_schema FAIL 0% 115 in → 75 out (55 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:none FAIL 0% 78 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.4:none__json_schema FAIL 0% 115 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.5:low FAIL 0% 79 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.5:none FAIL 0% 80 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-luna:high FAIL 0% 79 in → 57 out (45 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-luna:high__json_schema FAIL 0% 112 in → 77 out (57 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-luna:low FAIL 0% 78 in → 41 out (29 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-luna:max__json_schema FAIL 0% 110 in → 536 out (516 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-luna:medium FAIL 0% 77 in → 80 out (68 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:high FAIL 0% 74 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:high__json_schema FAIL 0% 116 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:low__json_schema FAIL 0% 114 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:medium FAIL 0% 81 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 113 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:none FAIL 0% 78 in → 10 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 117 in → 18 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) {"command":"git submodule status --recursive"}
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
poolside/laguna-m.1:none FAIL 0% 103 in → 7 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got ' git submodule status --recursive '
poolside/laguna-xs.2:xhigh FAIL 0% 137 in → 106 out (96 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got ' ``` git submodule status ``` '
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-06033a45-6983-9c1f-8c84-8d128afe3929","request_id":"06033a45-6983-9c1f-8c84-8d128afe3929"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e60f1a0d-6575-9fc7-b330-5f07154a0d32","request_id":"e60f1a0d-6575-9fc7-b330-5f07154a0d32"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-82229a3f-5229-9a84-9e4e-0144239d3d14","request_id":"82229a3f-5229-9a84-9e4e-0144239d3d14"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fdef1011-5a03-9b8d-b2e0-d17eb811a65d","request_id":"fdef1011-5a03-9b8d-b2e0-d17eb811a65d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-58184c50-0181-91c5-844a-52d9c746e2b2","request_id":"58184c50-0181-91c5-844a-52d9c746e2b2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-162d0cb8-531c-9ffb-adcc-ba304082ac18"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e02f9f04-97f9-901c-a44f-66fd1c0def60"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a83659bd-e428-9747-9a77-48b1cf94439f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-43349bd5-ab1d-9509-863f-79d514e0c57a"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6b00e85c-658a-9fbe-8e90-029035053c25"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9581a258-7f48-9e99-b181-9e5f08b4d6ff","request_id":"9581a258-7f48-9e99-b181-9e5f08b4d6ff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d697593a-4fab-9725-b3b3-64dd3881f1d0","request_id":"d697593a-4fab-9725-b3b3-64dd3881f1d0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dac4b6dd-429a-9ea2-ab2e-b225521c03a5","request_id":"dac4b6dd-429a-9ea2-ab2e-b225521c03a5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0516947d-0b3f-9a14-99ea-1eb964783a4a","request_id":"0516947d-0b3f-9a14-99ea-1eb964783a4a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-349835c0-9969-98f4-9933-1d3e93c50299","request_id":"349835c0-9969-98f4-9933-1d3e93c50299"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-112d5ea8-7dae-97b3-b04f-43ec0e7e2b57","request_id":"112d5ea8-7dae-97b3-b04f-43ec0e7e2b57"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b04adff6-6f9f-98ac-a8ca-6bc4d1e3344d","request_id":"b04adff6-6f9f-98ac-a8ca-6bc4d1e3344d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7561ed4c-1b83-9bfb-9801-5de712d451e0","request_id":"7561ed4c-1b83-9bfb-9801-5de712d451e0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-61e834f9-ddea-9583-8cd7-94226a309cd1","request_id":"61e834f9-ddea-9583-8cd7-94226a309cd1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f2fd8e12-42d4-9efa-8ce8-85ed36f9d021","request_id":"f2fd8e12-42d4-9efa-8ce8-85ed36f9d021"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:none FAIL 0% 95 in → 7 out (0 reasoning)
git submodule status --recursive
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
thinkingmachines/inkling-small:medium FAIL 0% 87 in → 7 out (0 reasoning)
git submodule status<|end_message|>
Failure: Expected 'git submodule status', got 'git submodule status<|end_message|>'
thinkingmachines/inkling-small:none FAIL 0% 85 in → 7 out (0 reasoning)
git submodule status<|end_message|>
Failure: Expected 'git submodule status', got 'git submodule status<|end_message|>'
Invalid JSON. Output: git submodule status{"command": "git submodule status"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule status
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 88 in → 7 out (0 reasoning)
git submodule status<|end_message|>
Failure: Expected 'git submodule status', got 'git submodule status<|end_message|>'
thinkingmachines/inkling:none__json_schema FAIL 0% 84 in → 22 out (0 reasoning)
git submodule status --recursive
JSON Schema Structured Output
(raw) { "command": "git submodule status --recursive" }
Failure: Expected 'git submodule status', got 'git submodule status --recursive'
x-ai/grok-4.5:low FAIL 0% 282 in → 180 out (173 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
x-ai/grok-4.5:medium FAIL 0% 284 in → 133 out (126 reasoning)
``` git submodule status ```
Failure: Expected 'git submodule status', got '``` git submodule status ```'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)