Add multiple submodules
Tests ability to add multiple submodules to a repository. Evaluates multi-submodule management.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 mkdir -p ../utils-src && cd ../utils-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'utils code' > utils.py && git add utils.py && git commit -m 'Initial utils'
  10. 10 git clone --bare ../utils-src ../utils-bare
  11. 11 git config protocol.file.allow always
Prompt
Add the repository at '../lib-bare' as submodule 'lib' and '../utils-bare' as submodule 'utils'. Output ONLY the commands to run, one per line.
Expected
git submodule add ../lib-bare lib git submodule add ../utils-bare utils

Scoped model quality, cost, API time, and token usage for submodule_usage/f007.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 74 in → 48 out (13 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-fable-5:high__json_schema PASS 100% 308 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-fable-5:low PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-fable-5:low__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-fable-5:max PASS 100% 74 in → 123 out (30 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-fable-5:max__json_schema PASS 100% 308 in → 154 out (36 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-fable-5:medium PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-fable-5:xhigh PASS 100% 74 in → 51 out (20 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 308 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-haiku-4.5:high PASS 100% 87 in → 160 out (113 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 272 in → 196 out (141 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-haiku-4.5:low PASS 100% 87 in → 166 out (110 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 272 in → 166 out (111 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-haiku-4.5:medium PASS 100% 87 in → 223 out (166 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 272 in → 163 out (107 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-haiku-4.5:none PASS 100% 58 in → 28 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 242 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 87 in → 163 out (108 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 272 in → 311 out (245 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:high PASS 100% 58 in → 87 out (31 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 243 in → 60 out (16 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:low PASS 100% 58 in → 29 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:max PASS 100% 58 in → 115 out (60 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 243 in → 127 out (63 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:medium PASS 100% 58 in → 29 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:none PASS 100% 58 in → 28 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 58 in → 87 out (31 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 243 in → 60 out (16 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:high PASS 100% 79 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.7:high__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:low PASS 100% 79 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.7:low__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:max PASS 100% 79 in → 38 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:medium PASS 100% 79 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:none PASS 100% 79 in → 38 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 79 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 313 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:high PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:low PASS 100% 74 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.8:low__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:max PASS 100% 74 in → 187 out (28 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:medium PASS 100% 74 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:none PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 308 in → 39 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-opus-4.8:xhigh PASS 100% 74 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:high PASS 100% 58 in → 117 out (62 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 243 in → 59 out (15 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:low PASS 100% 58 in → 60 out (19 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:max PASS 100% 58 in → 149 out (94 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 243 in → 155 out (92 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 58 in → 54 out (11 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:none PASS 100% 58 in → 28 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 243 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 58 in → 120 out (66 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 243 in → 58 out (14 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-5:high PASS 100% 74 in → 46 out (19 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
anthropic/claude-sonnet-5:high__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-5:low PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-5:medium PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-5:none PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 308 in → 39 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
anthropic/claude-sonnet-5:xhigh PASS 100% 74 in → 35 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 308 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
arcee-ai/trinity-large-thinking:high PASS 100% 52 in → 483 out (466 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
arcee-ai/trinity-large-thinking:low PASS 100% 52 in → 464 out (446 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
arcee-ai/trinity-large-thinking:medium PASS 100% 52 in → 339 out (320 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
arcee-ai/trinity-large-thinking:xhigh PASS 100% 52 in → 500 out (480 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
arcee-ai/trinity-mini:high__json_schema PASS 100% 52 in → 543 out (584 reasoning)
git submodule init git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands":[ "git submodule init", "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
arcee-ai/trinity-mini:low__json_schema PASS 100% 52 in → 584 out (653 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 52 in → 555 out (604 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 52 in → 414 out (434 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 127 in → 98 out (78 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 127 in → 163 out (133 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
deepseek/deepseek-v4-flash-0731:low PASS 100% 48 in → 237 out (231 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 48 in → 226 out (201 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
deepseek/deepseek-v4-flash-0731:max PASS 100% 140 in → 168 out (151 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 127 in → 117 out (88 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 48 in → 135 out (121 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 48 in → 170 out (139 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 48 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 48 in → 39 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 127 in → 196 out (191 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 48 in → 293 out (271 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash:high PASS 100% 48 in → 135 out (115 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash:low PASS 100% 48 in → 89 out (69 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash:medium PASS 100% 48 in → 112 out (91 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 48 in → 96 out (58 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash:none PASS 100% 48 in → 20 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 48 in → 34 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-flash:xhigh PASS 100% 127 in → 96 out (72 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 129 in → 228 out (193 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-pro:high PASS 100% 48 in → 216 out (195 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 246 in → 177 out (149 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
deepseek/deepseek-v4-pro:low PASS 100% 48 in → 217 out (197 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 246 in → 202 out (175 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
deepseek/deepseek-v4-pro:medium PASS 100% 48 in → 277 out (256 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 48 in → 34 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-pro:none PASS 100% 48 in → 20 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 48 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 48 in → 119 out (98 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 324 in → 456 out (428 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
google/gemini-3-flash-preview:high PASS 100% 43 in → 224 out (207 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3-flash-preview:high__json_schema PASS 100% 121 in → 397 out (376 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3-flash-preview:low PASS 100% 43 in → 188 out (171 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3-flash-preview:low__json_schema PASS 100% 121 in → 215 out (194 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3-flash-preview:medium PASS 100% 43 in → 164 out (147 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3-flash-preview:medium__json_schema PASS 100% 121 in → 241 out (220 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3-flash-preview:xhigh PASS 100% 43 in → 195 out (178 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 121 in → 263 out (242 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 43 in → 269 out (252 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 121 in → 623 out (602 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 43 in → 147 out (130 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 121 in → 134 out (111 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 44 in → 161 out (144 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 44 in → 192 out (171 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 44 in → 289 out (272 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 44 in → 812 out (791 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-pro-preview:high PASS 100% 43 in → 567 out (550 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 121 in → 412 out (391 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-pro-preview:low PASS 100% 43 in → 191 out (174 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 121 in → 99 out (63 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemini-3.1-pro-preview:medium PASS 100% 43 in → 321 out (304 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 121 in → 453 out (432 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 43 in → 451 out (434 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:high PASS 100% 43 in → 408 out (391 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 121 in → 448 out (427 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.5-flash-lite:low PASS 100% 44 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 44 in → 36 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemini-3.5-flash-lite:max PASS 100% 44 in → 386 out (369 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 44 in → 338 out (317 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.5-flash-lite:medium PASS 100% 43 in → 179 out (162 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 44 in → 363 out (342 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 44 in → 402 out (385 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 121 in → 622 out (601 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.5-flash:high PASS 100% 43 in → 375 out (358 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash:high__json_schema PASS 100% 121 in → 405 out (382 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
google/gemini-3.5-flash:low PASS 100% 43 in → 161 out (144 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash:low__json_schema PASS 100% 121 in → 140 out (104 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemini-3.5-flash:medium PASS 100% 43 in → 346 out (329 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash:medium__json_schema PASS 100% 121 in → 275 out (239 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemini-3.5-flash:xhigh PASS 100% 43 in → 392 out (375 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 121 in → 240 out (219 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.6-flash:high PASS 100% 44 in → 361 out (344 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.6-flash:high__json_schema PASS 100% 121 in → 337 out (316 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.6-flash:low PASS 100% 44 in → 241 out (224 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.6-flash:low__json_schema PASS 100% 121 in → 36 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemini-3.6-flash:max PASS 100% 44 in → 419 out (402 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.6-flash:max__json_schema PASS 100% 44 in → 414 out (393 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.6-flash:medium PASS 100% 44 in → 362 out (345 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.6-flash:medium__json_schema PASS 100% 44 in → 291 out (271 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemini-3.6-flash:xhigh PASS 100% 43 in → 266 out (249 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 44 in → 361 out (340 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
google/gemma-4-26b-a4b-it:high PASS 100% 59 in → 288 out (230 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 58 in → 278 out (224 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 59 in → 242 out (202 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-26b-a4b-it:medium PASS 100% 59 in → 175 out (127 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 59 in → 213 out (154 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 56 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 56 in → 38 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 59 in → 211 out (165 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 59 in → 202 out (140 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-31b-it:high PASS 100% 59 in → 161 out (128 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-31b-it:high__json_schema PASS 100% 59 in → 160 out (114 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
google/gemma-4-31b-it:low PASS 100% 59 in → 156 out (117 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-31b-it:low__json_schema PASS 100% 58 in → 232 out (197 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
google/gemma-4-31b-it:medium PASS 100% 62 in → 166 out (148 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-31b-it:medium__json_schema PASS 100% 58 in → 188 out (128 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-31b-it:none PASS 100% 56 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-31b-it:none__json_schema PASS 100% 56 in → 37 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
google/gemma-4-31b-it:xhigh PASS 100% 59 in → 207 out (162 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 59 in → 209 out (152 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
ibm-granite/granite-4.1-8b PASS 100% 52 in → 23 out
```bash git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
inclusionai/ling-2.6-flash PASS 100% 64 in → 20 out
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m2.5:high PASS 100% 80 in → 117 out (109 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.5:high__json_schema PASS 100% 80 in → 309 out (277 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m2.5:low PASS 100% 80 in → 116 out (97 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.5:low__json_schema PASS 100% 80 in → 154 out (143 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m2.5:medium PASS 100% 80 in → 471 out (538 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.5:medium__json_schema PASS 100% 83 in → 125 out (92 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m2.5:xhigh PASS 100% 80 in → 161 out (142 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 83 in → 151 out (119 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m2.7:high PASS 100% 83 in → 330 out (312 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.7:low PASS 100% 80 in → 335 out (316 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.7:low__json_schema PASS 100% 230 in → 489 out (465 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
minimax/minimax-m2.7:medium PASS 100% 83 in → 247 out (229 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.7:medium__json_schema PASS 100% 230 in → 349 out (325 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
minimax/minimax-m2.7:xhigh PASS 100% 83 in → 346 out (328 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 230 in → 298 out (265 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m3:high PASS 100% 215 in → 130 out (126 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
minimax/minimax-m3:high__json_schema PASS 100% 166 in → 32 out (31 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
minimax/minimax-m3:low PASS 100% 218 in → 220 out (235 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m3:low__json_schema PASS 100% 205 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
minimax/minimax-m3:medium PASS 100% 166 in → 40 out (21 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m3:medium__json_schema PASS 100% 205 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
minimax/minimax-m3:xhigh PASS 100% 218 in → 40 out (27 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
minimax/minimax-m3:xhigh__json_schema PASS 100% 166 in → 32 out (31 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
mistralai/devstral-2512 PASS 100% 47 in → 25 out
```bash git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
mistralai/devstral-2512__json_schema PASS 100% 47 in → 34 out
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
mistralai/mistral-medium-3-5:high PASS 100% 59 in → 250 out (247 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 47 in → 327 out (313 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
mistralai/mistral-medium-3-5:low PASS 100% 59 in → 372 out (376 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 47 in → 481 out (485 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
mistralai/mistral-medium-3-5:medium PASS 100% 59 in → 500 out (514 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 47 in → 249 out (231 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
mistralai/mistral-medium-3-5:none PASS 100% 59 in → 22 out (0 reasoning)
git submodule add '../lib-bare' lib git submodule add '../utils-bare' utils
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 47 in → 27 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 59 in → 411 out (431 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 47 in → 474 out (487 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
moonshotai/kimi-k2.6:high PASS 100% 50 in → 6,722 out (6,703 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.6:high__json_schema PASS 100% 49 in → 828 out (625 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
moonshotai/kimi-k2.6:low PASS 100% 50 in → 429 out (496 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.6:low__json_schema PASS 100% 49 in → 431 out (234 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
moonshotai/kimi-k2.6:medium PASS 100% 50 in → 225 out (206 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 50 in → 265 out (237 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"] }
moonshotai/kimi-k2.6:xhigh PASS 100% 49 in → 274 out (285 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 50 in → 472 out (518 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
moonshotai/kimi-k2.7-code:high PASS 100% 50 in → 232 out (249 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 143 in → 148 out (122 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
moonshotai/kimi-k2.7-code:low PASS 100% 49 in → 96 out (77 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 50 in → 135 out (124 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 49 in → 131 out (112 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 50 in → 183 out (180 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 50 in → 111 out (103 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 50 in → 345 out (359 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
moonshotai/kimi-k3:high PASS 100% 127 in → 657 out (625 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:high__json_schema PASS 100% 127 in → 236 out (204 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:low PASS 100% 127 in → 142 out (110 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:low__json_schema PASS 100% 127 in → 107 out (75 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:max PASS 100% 127 in → 127 out (92 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
moonshotai/kimi-k3:max__json_schema PASS 100% 127 in → 138 out (106 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:medium PASS 100% 127 in → 126 out (91 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 127 in → 313 out (281 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
moonshotai/kimi-k3:xhigh PASS 100% 127 in → 107 out (72 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 127 in → 110 out (78 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 60 in → 180 out (170 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 60 in → 199 out (170 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 60 in → 147 out (134 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 60 in → 1,447 out (1,598 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 60 in → 293 out (293 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 60 in → 213 out (184 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 60 in → 20 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 60 in → 36 out (0 reasoning)
git submodule add '../lib-bare' lib git submodule add '../utils-bare' utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add '../lib-bare' lib", "git submodule add '../utils-bare' utils" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 60 in → 154 out (147 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 60 in → 541 out (546 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 60 in → 174 out (142 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 60 in → 145 out (104 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 60 in → 63 out (48 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 60 in → 146 out (110 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 60 in → 67 out (43 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 60 in → 153 out (112 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 60 in → 20 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 60 in → 34 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 60 in → 181 out (176 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 60 in → 108 out (70 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
openai/gpt-5.4-mini:high PASS 100% 50 in → 202 out (177 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-mini:high__json_schema PASS 100% 96 in → 179 out (145 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-mini:low PASS 100% 50 in → 47 out (22 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-mini:low__json_schema PASS 100% 96 in → 113 out (79 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-mini:medium PASS 100% 50 in → 110 out (85 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-mini:medium__json_schema PASS 100% 96 in → 206 out (172 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-mini:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-mini:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-mini:xhigh PASS 100% 50 in → 154 out (129 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 96 in → 273 out (239 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-nano:high PASS 100% 50 in → 89 out (64 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-nano:high__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-nano:low PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-nano:low__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-nano:medium PASS 100% 50 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils git submodule update --init --recursive
openai/gpt-5.4-nano:medium__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4-nano:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-nano:none__json_schema PASS 100% 96 in → 41 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils git submodule update --init --recursive
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils","git submodule update --init --recursive"]}
openai/gpt-5.4-nano:xhigh PASS 100% 50 in → 228 out (203 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 96 in → 972 out (938 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4:high PASS 100% 50 in → 330 out (305 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4:high__json_schema PASS 100% 96 in → 365 out (317 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.4:low PASS 100% 50 in → 56 out (31 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4:low__json_schema PASS 100% 96 in → 64 out (30 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4:medium PASS 100% 50 in → 131 out (106 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4:medium__json_schema PASS 100% 96 in → 284 out (250 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.4:xhigh PASS 100% 50 in → 295 out (270 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.4:xhigh__json_schema PASS 100% 96 in → 14,874 out (14,826 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.5:high PASS 100% 50 in → 84 out (59 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.5:high__json_schema PASS 100% 96 in → 346 out (298 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.5:low PASS 100% 50 in → 81 out (56 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.5:low__json_schema PASS 100% 96 in → 65 out (31 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.5:medium PASS 100% 50 in → 98 out (73 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.5:medium__json_schema PASS 100% 96 in → 81 out (47 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.5:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.5:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.5:xhigh PASS 100% 50 in → 555 out (516 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
openai/gpt-5.5:xhigh__json_schema PASS 100% 96 in → 129 out (95 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:high PASS 100% 50 in → 97 out (72 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:high__json_schema PASS 100% 96 in → 94 out (60 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:low PASS 100% 50 in → 60 out (35 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:low__json_schema PASS 100% 96 in → 68 out (34 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:max PASS 100% 50 in → 112 out (87 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:max__json_schema PASS 100% 96 in → 450 out (416 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:medium PASS 100% 50 in → 72 out (47 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:medium__json_schema PASS 100% 96 in → 73 out (39 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-luna:xhigh PASS 100% 50 in → 81 out (56 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 96 in → 264 out (230 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:high PASS 100% 50 in → 73 out (48 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-sol:high__json_schema PASS 100% 96 in → 208 out (174 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:low PASS 100% 50 in → 58 out (33 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-sol:low__json_schema PASS 100% 96 in → 147 out (99 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:max PASS 100% 50 in → 100 out (75 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-sol:max__json_schema PASS 100% 96 in → 550 out (516 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:medium PASS 100% 50 in → 72 out (47 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-sol:medium__json_schema PASS 100% 96 in → 238 out (190 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-sol:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-sol:xhigh PASS 100% 50 in → 449 out (410 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 96 in → 685 out (637 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git -c protocol.file.allow=always submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git -c protocol.file.allow=always submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:high PASS 100% 50 in → 57 out (32 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:high__json_schema PASS 100% 96 in → 96 out (62 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:low PASS 100% 50 in → 55 out (30 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:low__json_schema PASS 100% 96 in → 73 out (39 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:max PASS 100% 50 in → 106 out (81 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:max__json_schema PASS 100% 96 in → 191 out (157 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:medium PASS 100% 50 in → 50 out (25 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:medium__json_schema PASS 100% 96 in → 98 out (64 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:none PASS 100% 50 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:none__json_schema PASS 100% 96 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-5.6-terra:xhigh PASS 100% 50 in → 60 out (35 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 96 in → 107 out (73 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-oss-120b:high PASS 100% 109 in → 657 out (705 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-120b:high__json_schema PASS 100% 98 in → 363 out (349 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-oss-120b:low PASS 100% 111 in → 33 out (5 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-120b:low__json_schema PASS 100% 109 in → 90 out (50 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
openai/gpt-oss-120b:medium PASS 100% 98 in → 131 out (109 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-120b:medium__json_schema PASS 100% 113 in → 114 out (85 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
openai/gpt-oss-120b:xhigh PASS 100% 111 in → 171 out (134 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 113 in → 591 out (623 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils git commit -m "Add lib and utils submodules"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils", "git commit -m \"Add lib and utils submodules\"" ] }
openai/gpt-oss-20b:high PASS 100% 111 in → 131 out (112 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-20b:low PASS 100% 111 in → 148 out (129 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-20b:low__json_schema PASS 100% 109 in → 89 out (52 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
openai/gpt-oss-20b:medium PASS 100% 111 in → 174 out (138 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-20b:medium__json_schema PASS 100% 111 in → 217 out (203 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
openai/gpt-oss-20b:xhigh PASS 100% 109 in → 544 out (573 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 109 in → 740 out (790 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands":[ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-m.1:high PASS 100% 56 in → 455 out (436 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-m.1:high__json_schema PASS 100% 56 in → 280 out (256 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-m.1:low PASS 100% 56 in → 392 out (373 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-m.1:low__json_schema PASS 100% 56 in → 774 out (751 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-m.1:medium PASS 100% 56 in → 377 out (358 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-m.1:medium__json_schema PASS 100% 56 in → 1,135 out (1,097 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-m.1:none PASS 100% 56 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-m.1:none__json_schema PASS 100% 56 in → 38 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils git commit -m 'Add lib and utils submodules'
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils", "git commit -m 'Add lib and utils submodules'"]}
poolside/laguna-m.1:xhigh PASS 100% 56 in → 1,800 out (1,770 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils git commit -m "Add lib and utils submodules"
poolside/laguna-m.1:xhigh__json_schema PASS 100% 56 in → 1,018 out (981 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-xs-2.1:high PASS 100% 56 in → 308 out (290 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs-2.1:high__json_schema PASS 100% 56 in → 403 out (366 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-xs-2.1:low PASS 100% 56 in → 335 out (318 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs-2.1:low__json_schema PASS 100% 56 in → 544 out (520 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs-2.1:medium PASS 100% 56 in → 297 out (280 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 56 in → 347 out (322 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs-2.1:none PASS 100% 56 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs-2.1:none__json_schema PASS 100% 56 in → 37 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 56 in → 331 out (314 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 56 in → 270 out (233 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
poolside/laguna-xs.2:high PASS 100% 93 in → 116 out (96 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs.2:high__json_schema PASS 100% 93 in → 120 out (95 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs.2:low PASS 100% 93 in → 153 out (134 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs.2:low__json_schema PASS 100% 93 in → 161 out (137 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs.2:medium PASS 100% 93 in → 119 out (99 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs.2:medium__json_schema PASS 100% 93 in → 139 out (115 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs.2:none PASS 100% 93 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs.2:none__json_schema PASS 100% 93 in → 24 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
poolside/laguna-xs.2:xhigh PASS 100% 93 in → 133 out (113 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 93 in → 168 out (131 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-27b:high PASS 100% 53 in → 930 out (931 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-27b:high__json_schema PASS 100% 53 in → 324 out (269 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-27b:low PASS 100% 53 in → 801 out (804 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-27b:low__json_schema PASS 100% 53 in → 404 out (369 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-27b:medium PASS 100% 53 in → 1,347 out (1,351 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-27b:medium__json_schema PASS 100% 53 in → 1,137 out (1,126 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-27b:none PASS 100% 55 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-27b:none__json_schema PASS 100% 55 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
qwen/qwen3.6-27b:xhigh PASS 100% 53 in → 379 out (357 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 53 in → 852 out (1 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 53 in → 876 out (845 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 53 in → 903 out (877 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 53 in → 407 out (385 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 53 in → 884 out (854 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 53 in → 327 out (297 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 53 in → 826 out (805 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 55 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 55 in → 41 out (0 reasoning)
git submodule add '../lib-bare' 'lib' git submodule add '../utils-bare' 'utils'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add '../lib-bare' 'lib'", "git submodule add '../utils-bare' 'utils'" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 53 in → 922 out (941 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 53 in → 782 out (763 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
qwen/qwen3.6-flash:high PASS 100% 53 in → 692 out (670 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-flash:low PASS 100% 53 in → 966 out (944 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-flash:medium PASS 100% 53 in → 981 out (959 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-flash:none PASS 100% 55 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.6-flash:xhigh PASS 100% 53 in → 1,049 out (1,027 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:high PASS 100% 53 in → 1,087 out (1,065 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:low PASS 100% 53 in → 977 out (955 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:max PASS 100% 53 in → 824 out (802 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:medium PASS 100% 53 in → 826 out (804 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:none PASS 100% 55 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-flash:xhigh PASS 100% 53 in → 303 out (281 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-max:high PASS 100% 53 in → 289 out (267 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-max:low PASS 100% 53 in → 395 out (373 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-max:medium PASS 100% 53 in → 322 out (300 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-max:none PASS 100% 55 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-max:xhigh PASS 100% 53 in → 168 out (146 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-plus:high PASS 100% 53 in → 508 out (486 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-plus:low PASS 100% 53 in → 636 out (614 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-plus:medium PASS 100% 53 in → 628 out (606 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-plus:none PASS 100% 55 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
qwen/qwen3.7-plus:xhigh PASS 100% 53 in → 569 out (547 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:high PASS 100% 54 in → 549 out (530 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:high__json_schema PASS 100% 54 in → 321 out (304 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
tencent/hy3:low PASS 100% 54 in → 305 out (286 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:low__json_schema PASS 100% 54 in → 451 out (430 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ]}
tencent/hy3:medium PASS 100% 54 in → 333 out (314 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:medium__json_schema PASS 100% 54 in → 252 out (232 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
tencent/hy3:none PASS 100% 57 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:none__json_schema PASS 100% 57 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
tencent/hy3:xhigh PASS 100% 54 in → 462 out (443 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
tencent/hy3:xhigh__json_schema PASS 100% 54 in → 451 out (437 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling-small:high PASS 100% 59 in → 170 out (148 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:high__json_schema PASS 100% 59 in → 156 out (117 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling-small:low PASS 100% 59 in → 104 out (79 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:low__json_schema PASS 100% 59 in → 87 out (48 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling-small:max PASS 100% 59 in → 249 out (224 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:max__json_schema PASS 100% 59 in → 251 out (212 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling-small:medium PASS 100% 59 in → 154 out (130 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:medium__json_schema PASS 100% 59 in → 134 out (96 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling-small:none PASS 100% 57 in → 22 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:xhigh PASS 100% 59 in → 156 out (131 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 59 in → 189 out (151 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling:high PASS 100% 59 in → 115 out (89 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:low PASS 100% 59 in → 47 out (22 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:max PASS 100% 59 in → 196 out (171 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:max__json_schema PASS 100% 59 in → 156 out (118 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
thinkingmachines/inkling:medium PASS 100% 59 in → 62 out (38 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:none PASS 100% 57 in → 22 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:xhigh PASS 100% 59 in → 123 out (97 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
thinkingmachines/inkling:xhigh__json_schema PASS 100% 59 in → 133 out (95 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.3:high PASS 100% 227 in → 156 out (134 reasoning)
```bash git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
x-ai/grok-4.3:high__json_schema PASS 100% 293 in → 435 out (414 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
x-ai/grok-4.3:low PASS 100% 233 in → 219 out (200 reasoning)
git submodule add '../lib-bare' lib git submodule add '../utils-bare' utils
x-ai/grok-4.3:low__json_schema PASS 100% 299 in → 355 out (324 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.3:max PASS 100% 227 in → 157 out (140 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.3:max__json_schema PASS 100% 293 in → 402 out (371 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.3:medium PASS 100% 233 in → 326 out (309 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.3:medium__json_schema PASS 100% 299 in → 317 out (286 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.3:none PASS 100% 225 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.3:none__json_schema PASS 100% 291 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
x-ai/grok-4.3:xhigh PASS 100% 227 in → 180 out (163 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.3:xhigh__json_schema PASS 100% 293 in → 378 out (347 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.5:high PASS 100% 248 in → 161 out (144 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.5:high__json_schema PASS 100% 327 in → 259 out (227 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.5:low PASS 100% 248 in → 131 out (114 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.5:low__json_schema PASS 100% 327 in → 290 out (258 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
x-ai/grok-4.5:max PASS 100% 248 in → 184 out (167 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
x-ai/grok-4.5:max__json_schema PASS 100% 327 in → 310 out (288 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
x-ai/grok-4.5:medium PASS 100% 248 in → 143 out (122 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
x-ai/grok-4.5:medium__json_schema PASS 100% 327 in → 297 out (275 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
x-ai/grok-4.5:xhigh PASS 100% 248 in → 157 out (136 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 327 in → 265 out (233 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
z-ai/glm-4.7-flash:high PASS 100% 47 in → 609 out (572 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-4.7-flash:low PASS 100% 47 in → 504 out (485 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-4.7-flash:medium PASS 100% 47 in → 624 out (592 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 47 in → 1,592 out (1,594 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
z-ai/glm-4.7-flash:none PASS 100% 47 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-4.7-flash:none__json_schema PASS 100% 47 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
z-ai/glm-4.7-flash:xhigh PASS 100% 47 in → 448 out (432 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-5.2:high PASS 100% 54 in → 198 out (179 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-5.2:high__json_schema PASS 100% 54 in → 254 out (229 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
z-ai/glm-5.2:low PASS 100% 54 in → 227 out (208 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-5.2:low__json_schema PASS 100% 239 in → 55 out (32 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
z-ai/glm-5.2:medium PASS 100% 54 in → 193 out (167 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-5.2:medium__json_schema PASS 100% 54 in → 237 out (210 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"] }
z-ai/glm-5.2:none PASS 100% 48 in → 22 out (0 reasoning)
``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ```
z-ai/glm-5.2:none__json_schema PASS 100% 48 in → 24 out (0 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
z-ai/glm-5.2:xhigh PASS 100% 54 in → 335 out (319 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
z-ai/glm-5.2:xhigh__json_schema PASS 100% 47 in → 352 out (321 reasoning)
git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils" ] }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 52 in → 472 out (511 reasoning)
git submodule add ../lib-barelib git submodule add ../utils-bare utils
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
arcee-ai/trinity-mini:low FAIL 0% 52 in → 266 out (271 reasoning)
git submodule add../lib-bare lib git submodule add ../utils-bare utils
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
arcee-ai/trinity-mini:medium FAIL 0% 52 in → 302 out (310 reasoning)
git submodule add ../lib-barelib git submodule add ../utils-bare utils
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 52 in → 380 out (406 reasoning)
git submodule initgit submodule add ../lib-bare lib git submodule add ../utils-bare utils
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
Invalid JSON. Output: gitsubmodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
git submodule add ../lib- bare lib git submodule add ../utils- bare utils
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib- bare lib", "git submodule add ../utils- bare utils" ] }
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
liquid/lfm-2-24b-a2b FAIL 0% 55 in → 136 out
To add the specified repositories as submodules in your Git repository, follow these steps: 1. Navigate to your project directory (if not already there). 2. Run the following commands: ``` git submodule add ../lib-bare lib git submodule add ../utils-bare utils ``` These commands will initialize and commit the two submodules into your repository under the names `lib` and `utils` respectively. Make sure you are in the root of your Git repository when running these commands. After executing them, you'll need to commit the changes with `git commit -m "Added submodules lib and utils"`.
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
Invalid JSON. Output: To add the specified repositories as submodules in your Git repository, follow these steps: 1. Navigate to your main repository directory. 2. Run `git submodule add ../lib-bare lib` to add the `lib-bare` repository as a submodule named `lib`. 3. Run `git submodule add ../utils-bare util` to add the `utils-bare` repository as a submodule named `utils`. These commands will initialize both submodules within your project structure. git submodule add ../lib-bare lib git submodule add ../utils-bare util
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
openai/gpt-oss-20b:high__json_schema FAIL 0% 111 in → 26 out (0 reasoning)
[...]
JSON Schema Structured Output
(raw) { "commands": [ "[...]" ] }
Failure: Failed assertions: [{'type': 'dir_exists', 'path': 'lib'}, {'type': 'dir_exists', 'path': 'utils'}, {'type': 'file_content', 'path': 'lib/lib.py', 'value': 'lib code'}, {'type': 'file_content', 'path': 'utils/utils.py', 'value': 'utils code'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-359ea221-4a6c-95a9-a1b5-4ae7b3d349a5","request_id":"359ea221-4a6c-95a9-a1b5-4ae7b3d349a5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d441c11a-124f-94e7-bada-46f874e43163","request_id":"d441c11a-124f-94e7-bada-46f874e43163"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fa046a12-d893-9bb0-919e-82a1fdf9dc53","request_id":"fa046a12-d893-9bb0-919e-82a1fdf9dc53"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4c7aae75-e8dd-93ff-a6b2-399ffcb32b43","request_id":"4c7aae75-e8dd-93ff-a6b2-399ffcb32b43"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-678b9f53-6057-9143-bc69-2fd2e0f58691","request_id":"678b9f53-6057-9143-bc69-2fd2e0f58691"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7fc2f2e8-b737-93c7-b1b8-9b3fc68c9e8b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4bb74904-8c24-97da-b8e7-1f734bec9c4c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-60ae20f5-c176-9a47-84a1-8890a41d0a59"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-686c55f8-be7a-93e6-934d-af8e9fb0fbd9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-778b72b3-1891-9382-9d5b-0e2a9643feb7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dbe2cc7e-10fc-9cda-9c86-7abc017d732c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9cc290b1-665f-91a0-b137-c3cad45b7bd4","request_id":"9cc290b1-665f-91a0-b137-c3cad45b7bd4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c5e00985-20ca-9adb-a990-f887c201cd54","request_id":"c5e00985-20ca-9adb-a990-f887c201cd54"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-33c53478-bac3-94f6-a9d8-99f9778e8804","request_id":"33c53478-bac3-94f6-a9d8-99f9778e8804"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c43bb734-3c96-908b-91df-056691408100","request_id":"c43bb734-3c96-908b-91df-056691408100"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b49124a6-46e5-90a2-8927-69fb6417ae93","request_id":"b49124a6-46e5-90a2-8927-69fb6417ae93"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-cc8eeb7d-2e9a-9207-904c-9381e37500f1","request_id":"cc8eeb7d-2e9a-9207-904c-9381e37500f1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-436d8b8c-85b1-9370-b606-a3fd6e83c6c1","request_id":"436d8b8c-85b1-9370-b606-a3fd6e83c6c1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c2397951-1f6b-9a90-aa36-68c9905f51c6","request_id":"c2397951-1f6b-9a90-aa36-68c9905f51c6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1fded7ae-65eb-9845-b7c6-74f52c3b247f","request_id":"1fded7ae-65eb-9845-b7c6-74f52c3b247f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-022e9747-7ff6-9e6d-89e4-80e2053f6e8f","request_id":"022e9747-7ff6-9e6d-89e4-80e2053f6e8f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils{"commands": ["git submodule add ../lib-bare lib", "git submodule add ../utils-bare utils"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib git submodule add ../utils-bare utils{"commands":["git submodule add ../lib-bare lib","git submodule add ../utils-bare utils"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)