f005 — submodule_usage

← submodule_usage
Add submodule and commit
Tests ability to add a submodule and commit the change. Evaluates end-to-end submodule-addition workflow.
easy submodule add commit workflow
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
Prompt
Add the repository at '../lib-bare' as a git submodule at path 'lib', then add and commit the changes. Output ONLY the commands to run, one per line.
Expected
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m 'Add lib submodule'
Fixture Metrics

Scoped model quality, cost, API time, and token usage for submodule_usage/f005.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 68 in → 82 out (28 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-fable-5:high__json_schema PASS 100% 302 in → 55 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
anthropic/claude-fable-5:low PASS 100% 68 in → 54 out (19 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-fable-5:low__json_schema PASS 100% 302 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
anthropic/claude-fable-5:max PASS 100% 68 in → 444 out (92 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-fable-5:max__json_schema PASS 100% 302 in → 265 out (64 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule at path lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule at path lib\""]}
anthropic/claude-fable-5:medium PASS 100% 68 in → 77 out (25 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 302 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
anthropic/claude-fable-5:xhigh PASS 100% 68 in → 174 out (39 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule at path lib" ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 302 in → 105 out (36 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule at path lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule at path lib\""]}
anthropic/claude-haiku-4.5:high PASS 100% 84 in → 562 out (479 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 269 in → 354 out (263 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
anthropic/claude-haiku-4.5:low PASS 100% 84 in → 327 out (240 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 269 in → 317 out (234 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
anthropic/claude-haiku-4.5:medium PASS 100% 84 in → 326 out (255 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 269 in → 416 out (319 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
anthropic/claude-haiku-4.5:none PASS 100% 55 in → 43 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at path lib" ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 239 in → 41 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m 'Add lib submodule'"]}
anthropic/claude-haiku-4.5:xhigh PASS 100% 84 in → 325 out (246 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 269 in → 876 out (530 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
anthropic/claude-opus-4.6:high PASS 100% 55 in → 88 out (28 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 240 in → 82 out (21 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule 'lib' from '../lib-bare'"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add submodule 'lib' from '../lib-bare'\""]}
anthropic/claude-opus-4.6:low PASS 100% 55 in → 38 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.6:max PASS 100% 55 in → 88 out (28 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 240 in → 134 out (71 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at path lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at path lib\""]}
anthropic/claude-opus-4.6:medium PASS 100% 55 in → 38 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.6:none PASS 100% 55 in → 42 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.6:xhigh PASS 100% 55 in → 87 out (26 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 240 in → 82 out (21 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule 'lib' from '../lib-bare'"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add submodule 'lib' from '../lib-bare'\""]}
anthropic/claude-opus-4.7:high PASS 100% 73 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-opus-4.7:high__json_schema PASS 100% 307 in → 61 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.7:low PASS 100% 73 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-opus-4.7:low__json_schema PASS 100% 307 in → 55 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
anthropic/claude-opus-4.7:max PASS 100% 73 in → 754 out (79 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 307 in → 61 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.7:medium PASS 100% 73 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 307 in → 61 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.7:none PASS 100% 73 in → 56 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 307 in → 61 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.7:xhigh PASS 100% 73 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 307 in → 61 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:high PASS 100% 68 in → 77 out (25 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare repository as submodule at lib" ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 302 in → 59 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:low PASS 100% 68 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare repository as submodule at lib"
anthropic/claude-opus-4.8:low__json_schema PASS 100% 302 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:max PASS 100% 68 in → 350 out (69 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 302 in → 239 out (69 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:medium PASS 100% 68 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare repository as submodule at lib"
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 302 in → 59 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:none PASS 100% 68 in → 53 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 302 in → 58 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-opus-4.8:xhigh PASS 100% 68 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare repository as submodule at lib"
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 302 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-4.6:high PASS 100% 55 in → 86 out (27 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-4.6:low PASS 100% 55 in → 60 out (7 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-4.6:max PASS 100% 55 in → 154 out (83 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 240 in → 160 out (90 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
anthropic/claude-sonnet-4.6:medium PASS 100% 55 in → 86 out (26 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-4.6:none PASS 100% 55 in → 42 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 55 in → 86 out (27 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 240 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-5:high PASS 100% 68 in → 53 out (0 reasoning)
``` git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 302 in → 59 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-5:low PASS 100% 68 in → 51 out (15 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
anthropic/claude-sonnet-5:low__json_schema PASS 100% 302 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
anthropic/claude-sonnet-5:medium PASS 100% 68 in → 49 out (0 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare as submodule at lib"
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 302 in → 59 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
anthropic/claude-sonnet-5:none PASS 100% 68 in → 53 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 302 in → 57 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare submodule at lib\""]}
anthropic/claude-sonnet-5:xhigh PASS 100% 68 in → 53 out (0 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 302 in → 59 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule at lib\""]}
arcee-ai/trinity-large-thinking:high PASS 100% 52 in → 1,123 out (1,099 reasoning)
git submodule add ../lib-bare lib git add .gitmodules git commit -m "Add lib submodule"
arcee-ai/trinity-large-thinking:low PASS 100% 52 in → 697 out (678 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as submodule"
arcee-ai/trinity-large-thinking:medium PASS 100% 52 in → 1,236 out (1,216 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib as submodule"
arcee-ai/trinity-large-thinking:xhigh PASS 100% 52 in → 2,192 out (2,165 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a submodule"
arcee-ai/trinity-mini:high__json_schema PASS 100% 52 in → 464 out (508 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add lib submodule\"" ] }
arcee-ai/trinity-mini:low PASS 100% 52 in → 365 out (391 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
arcee-ai/trinity-mini:low__json_schema PASS 100% 52 in → 812 out (872 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib-bare"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add submodule lib-bare\"" ] }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 52 in → 448 out (463 reasoning)
git submodule add ../lib-bare lib git add .gitmodules git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules", "git commit -m \"Add lib submodule\"" ] }
deepseek/deepseek-v4-flash-0731:high PASS 100% 126 in → 255 out (227 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 47 in → 390 out (358 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add lib submodule\"" ] }
deepseek/deepseek-v4-flash-0731:low PASS 100% 47 in → 308 out (298 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 47 in → 296 out (257 reasoning)
git submodule add ../lib-bare lib git add . git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m 'Add lib submodule'" ] }
deepseek/deepseek-v4-flash-0731:max PASS 100% 126 in → 173 out (147 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 126 in → 231 out (207 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 47 in → 520 out (499 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 47 in → 332 out (286 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
deepseek/deepseek-v4-flash-0731:none PASS 100% 47 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a git submodule"
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 47 in → 45 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as submodule\"" ] }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 126 in → 325 out (312 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 47 in → 552 out (513 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
deepseek/deepseek-v4-flash:high PASS 100% 47 in → 251 out (241 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 49 in → 587 out (539 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a git submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as a git submodule\"" ] }
deepseek/deepseek-v4-flash:low PASS 100% 47 in → 365 out (343 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 49 in → 228 out (177 reasoning)
git submodule add ../lib-bare ./lib git add .gitmodules lib git commit -m "Add submodule ../lib-bare at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare ./lib", "git add .gitmodules lib", "git commit -m \"Add submodule ../lib-bare at lib\"" ] }
deepseek/deepseek-v4-flash:medium PASS 100% 47 in → 144 out (119 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 49 in → 289 out (239 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule at lib\"" ] }
deepseek/deepseek-v4-flash:none PASS 100% 47 in → 32 out (0 reasoning)
```bash git submodule add ../lib-bare lib git add lib git commit -m "Add lib as a git submodule" ```
deepseek/deepseek-v4-flash:xhigh PASS 100% 126 in → 496 out (515 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule lib"
deepseek/deepseek-v4-pro:high PASS 100% 47 in → 448 out (419 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 245 in → 262 out (233 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add submodule lib\""]}
deepseek/deepseek-v4-pro:low PASS 100% 47 in → 347 out (332 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 49 in → 385 out (349 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add submodule lib\"" ] }
deepseek/deepseek-v4-pro:medium PASS 100% 47 in → 265 out (231 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 245 in → 149 out (112 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add lib .gitmodules", "git commit -m \"Add lib submodule\""]}
deepseek/deepseek-v4-pro:none PASS 100% 47 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 47 in → 50 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
deepseek/deepseek-v4-pro:xhigh PASS 100% 126 in → 562 out (540 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 47 in → 50 out (49 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
google/gemini-3-flash-preview:high PASS 100% 43 in → 1,878 out (1,851 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule"
google/gemini-3-flash-preview:high__json_schema PASS 100% 44 in → 153 out (124 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
google/gemini-3-flash-preview:low PASS 100% 43 in → 238 out (214 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule at lib"
google/gemini-3-flash-preview:low__json_schema PASS 100% 121 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as a submodule\""]}
google/gemini-3-flash-preview:medium PASS 100% 43 in → 877 out (853 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
google/gemini-3-flash-preview:medium__json_schema PASS 100% 121 in → 454 out (418 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at path lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as a submodule at path lib\""]}
google/gemini-3-flash-preview:xhigh PASS 100% 43 in → 1,187 out (1,157 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a git submodule at path lib"
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 121 in → 430 out (395 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule at path 'lib'"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib-bare as a submodule at path 'lib'\""]}
google/gemini-3.1-flash-lite-preview:high PASS 100% 44 in → 435 out (412 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule at lib"
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 44 in → 722 out (700 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 44 in → 152 out (130 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 121 in → 167 out (120 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add submodule lib\"" ] }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 44 in → 595 out (577 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 44 in → 342 out (305 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add submodule lib\"" ] }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 43 in → 373 out (348 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 44 in → 1,082 out (1,056 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
google/gemini-3.1-pro-preview:high PASS 100% 43 in → 705 out (687 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 121 in → 498 out (476 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.1-pro-preview:low PASS 100% 43 in → 393 out (375 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 121 in → 321 out (299 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.1-pro-preview:medium PASS 100% 43 in → 438 out (420 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 121 in → 615 out (593 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add submodule lib\""]}
google/gemini-3.1-pro-preview:xhigh PASS 100% 43 in → 1,135 out (1,117 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 121 in → 550 out (528 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.5-flash-lite:high PASS 100% 43 in → 828 out (803 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 121 in → 399 out (373 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add submodule lib\""]}
google/gemini-3.5-flash-lite:low PASS 100% 44 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 44 in → 44 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
google/gemini-3.5-flash-lite:max PASS 100% 44 in → 1,561 out (1,543 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 44 in → 508 out (482 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
google/gemini-3.5-flash-lite:medium PASS 100% 44 in → 289 out (264 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 121 in → 457 out (431 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
google/gemini-3.5-flash-lite:xhigh PASS 100% 44 in → 422 out (397 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 121 in → 545 out (516 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\""]}
google/gemini-3.5-flash:high PASS 100% 43 in → 1,878 out (1,860 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
google/gemini-3.5-flash:high__json_schema PASS 100% 121 in → 674 out (652 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.5-flash:low PASS 100% 43 in → 220 out (195 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash:low__json_schema PASS 100% 121 in → 231 out (178 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
google/gemini-3.5-flash:medium PASS 100% 43 in → 896 out (871 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.5-flash:medium__json_schema PASS 100% 121 in → 589 out (542 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
google/gemini-3.5-flash:xhigh PASS 100% 43 in → 854 out (825 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule at lib"
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 121 in → 566 out (536 reasoning)
git submodule add ../lib-bare lib git commit -m 'Add lib-bare as submodule at path lib'
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m 'Add lib-bare as submodule at path lib'"]}
google/gemini-3.6-flash:high PASS 100% 44 in → 1,120 out (1,095 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.6-flash:high__json_schema PASS 100% 44 in → 553 out (524 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.6-flash:low PASS 100% 44 in → 389 out (364 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
google/gemini-3.6-flash:low__json_schema PASS 100% 44 in → 27 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a git submodule at path lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a git submodule at path lib\"" ] }
google/gemini-3.6-flash:max PASS 100% 43 in → 923 out (901 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
google/gemini-3.6-flash:max__json_schema PASS 100% 121 in → 460 out (405 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a git submodule at path lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a git submodule at path lib\"" ] }
google/gemini-3.6-flash:medium PASS 100% 44 in → 744 out (722 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
google/gemini-3.6-flash:medium__json_schema PASS 100% 121 in → 411 out (382 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
google/gemini-3.6-flash:xhigh PASS 100% 44 in → 948 out (926 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 121 in → 479 out (444 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add ../lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add ../lib-bare as submodule at lib\""]}
google/gemma-4-26b-a4b-it:high PASS 100% 59 in → 630 out (603 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 58 in → 1,068 out (975 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare submodule\"" ] }
google/gemma-4-26b-a4b-it:low PASS 100% 59 in → 660 out (584 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 59 in → 601 out (515 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
google/gemma-4-26b-a4b-it:medium PASS 100% 59 in → 1,402 out (1,344 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 59 in → 1,031 out (941 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule\"" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 56 in → 32 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule in lib"
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 56 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule at path lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule at path lib\"" ] }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 59 in → 405 out (353 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 59 in → 1,009 out (902 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
google/gemma-4-31b-it:high PASS 100% 59 in → 245 out (187 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-31b-it:high__json_schema PASS 100% 59 in → 850 out (759 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare submodule\"" ] }
google/gemma-4-31b-it:low PASS 100% 59 in → 280 out (222 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-31b-it:low__json_schema PASS 100% 58 in → 238 out (176 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
google/gemma-4-31b-it:medium PASS 100% 62 in → 258 out (231 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-31b-it:medium__json_schema PASS 100% 58 in → 1,222 out (1,132 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
google/gemma-4-31b-it:none PASS 100% 56 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at path lib"
google/gemma-4-31b-it:none__json_schema PASS 100% 56 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a submodule\"" ] }
google/gemma-4-31b-it:xhigh PASS 100% 59 in → 259 out (198 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 59 in → 285 out (210 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare submodule\"" ] }
ibm-granite/granite-4.1-8b__json_schema PASS 100% 52 in → 39 out
git submodule add ../lib-bare lib git add lib git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib", "git commit -m 'Add lib submodule'" ] }
inclusionai/ling-2.6-flash PASS 100% 63 in → 30 out
git submodule add ../lib-bare lib git add lib git commit -m "Add lib-bare as submodule at lib"
inclusionai/ling-2.6-flash__json_schema PASS 100% 63 in → 41 out
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule at lib\""]}
liquid/lfm-2-24b-a2b PASS 100% 54 in → 31 out
git submodule add ../lib-bare lib git commit -m "Added submodule from ../lib-bare at path 'lib'"
minimax/minimax-m2.5:high PASS 100% 80 in → 1,009 out (982 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
minimax/minimax-m2.5:high__json_schema PASS 100% 80 in → 174 out (138 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib", "git commit -m \"Add lib as submodule\"" ] }
minimax/minimax-m2.5:low PASS 100% 80 in → 335 out (310 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as a submodule"
minimax/minimax-m2.5:low__json_schema PASS 100% 80 in → 395 out (352 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
minimax/minimax-m2.5:medium__json_schema PASS 100% 83 in → 896 out (853 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
minimax/minimax-m2.5:xhigh PASS 100% 80 in → 594 out (678 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as submodule"
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 83 in → 187 out (151 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib as submodule\"" ] }
minimax/minimax-m2.7:high PASS 100% 83 in → 532 out (506 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
minimax/minimax-m2.7:high__json_schema PASS 100% 230 in → 717 out (674 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
minimax/minimax-m2.7:low PASS 100% 80 in → 792 out (769 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
minimax/minimax-m2.7:low__json_schema PASS 100% 80 in → 42 out (41 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib", "git commit -m \"Add lib as a submodule\"" ] }
minimax/minimax-m2.7:medium PASS 100% 80 in → 578 out (555 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
minimax/minimax-m2.7:medium__json_schema PASS 100% 230 in → 302 out (270 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add submodule lib\""]}
minimax/minimax-m2.7:xhigh PASS 100% 80 in → 533 out (587 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule lib"
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 230 in → 495 out (455 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
minimax/minimax-m3:high PASS 100% 218 in → 202 out (196 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
minimax/minimax-m3:high__json_schema PASS 100% 166 in → 38 out (37 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule at lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare submodule at lib\""]}
minimax/minimax-m3:low__json_schema PASS 100% 205 in → 38 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
minimax/minimax-m3:medium PASS 100% 215 in → 164 out (150 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
minimax/minimax-m3:xhigh PASS 100% 218 in → 53 out (24 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at lib"
minimax/minimax-m3:xhigh__json_schema PASS 100% 205 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare repository as submodule at path 'lib'"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare repository as submodule at path 'lib'\"" ] }
mistralai/devstral-2512 PASS 100% 46 in → 37 out
```bash git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" ```
mistralai/devstral-2512__json_schema PASS 100% 46 in → 44 out
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add submodule lib\"" ] }
mistralai/mistral-medium-3-5:high PASS 100% 58 in → 793 out (842 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 46 in → 1,950 out (2,100 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib submodule\""]}
mistralai/mistral-medium-3-5:low PASS 100% 58 in → 1,049 out (1,147 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 46 in → 278 out (240 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
mistralai/mistral-medium-3-5:medium PASS 100% 58 in → 592 out (624 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 46 in → 342 out (316 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
mistralai/mistral-medium-3-5:none PASS 100% 58 in → 40 out (0 reasoning)
```bash git submodule add '../lib-bare' lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at path 'lib'" ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 46 in → 44 out (0 reasoning)
git submodule add '../lib-bare' lib git add .gitmodules lib git commit -m "Add submodule lib-bare at path 'lib'"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add '../lib-bare' lib", "git add .gitmodules lib", "git commit -m \"Add submodule lib-bare at path 'lib'\""]}
mistralai/mistral-medium-3-5:xhigh PASS 100% 58 in → 975 out (1,021 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 46 in → 1,497 out (1,610 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib submodule\""]}
moonshotai/kimi-k2.6:high PASS 100% 50 in → 1,708 out (1,886 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.6:low PASS 100% 50 in → 2,900 out (3,079 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.6:low__json_schema PASS 100% 49 in → 1,484 out (979 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""] }
moonshotai/kimi-k2.6:medium PASS 100% 50 in → 666 out (672 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 50 in → 1,046 out (1,008 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m 'Add lib submodule'" ] }
moonshotai/kimi-k2.6:xhigh PASS 100% 50 in → 1,584 out (1,557 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 50 in → 1,429 out (1,392 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ]}
moonshotai/kimi-k2.7-code:high PASS 100% 49 in → 212 out (185 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.7-code:low PASS 100% 49 in → 1,108 out (1,077 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule at lib"
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 50 in → 144 out (110 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
moonshotai/kimi-k2.7-code:medium PASS 100% 50 in → 276 out (249 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 50 in → 358 out (321 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 50 in → 137 out (117 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 50 in → 462 out (411 reasoning)
git submodule add ../lib-bare lib git submodule update --init git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git submodule update --init", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
moonshotai/kimi-k3:high PASS 100% 127 in → 497 out (457 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:high__json_schema PASS 100% 233 in → 197 out (153 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
moonshotai/kimi-k3:low PASS 100% 127 in → 303 out (263 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:low__json_schema PASS 100% 127 in → 253 out (213 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:max PASS 100% 127 in → 256 out (216 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:max__json_schema PASS 100% 127 in → 215 out (175 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:medium PASS 100% 127 in → 193 out (153 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:medium__json_schema PASS 100% 127 in → 171 out (131 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
moonshotai/kimi-k3:xhigh PASS 100% 127 in → 242 out (200 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 127 in → 239 out (197 reasoning)
``` git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" ```
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 59 in → 160 out (136 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib submodule from ../lib-bare"
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 59 in → 3,044 out (3,411 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 59 in → 295 out (289 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib submodule"
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 59 in → 268 out (243 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib", "git commit -m \"Add lib submodule\"" ] }
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 59 in → 1,832 out (2,019 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 59 in → 29 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 59 in → 46 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule at path lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare submodule at path lib\"" ] }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 59 in → 270 out (260 reasoning)
git submodule add '../lib-bare' lib git add lib git commit -m "Add lib submodule from ../lib-bare"
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 59 in → 3,713 out (4,221 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add submodule lib\"" ] }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 59 in → 102 out (68 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 59 in → 148 out (96 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule\"" ] }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 59 in → 166 out (152 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 59 in → 184 out (131 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule\"" ] }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 59 in → 119 out (98 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 59 in → 114 out (64 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule\"" ] }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 59 in → 33 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 59 in → 49 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule at lib\"" ] }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 59 in → 134 out (97 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule at lib"
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 59 in → 142 out (87 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule at lib\"" ] }
openai/gpt-5.4-mini:high PASS 100% 49 in → 303 out (270 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-mini:high__json_schema PASS 100% 95 in → 232 out (190 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4-mini:low PASS 100% 49 in → 83 out (50 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-mini:low__json_schema PASS 100% 95 in → 118 out (76 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4-mini:medium PASS 100% 49 in → 157 out (124 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-mini:medium__json_schema PASS 100% 95 in → 282 out (240 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4-mini:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-mini:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4-mini:xhigh PASS 100% 49 in → 549 out (516 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 95 in → 14,697 out (14,655 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m 'Add lib submodule'"]}
openai/gpt-5.4-nano:high PASS 100% 49 in → 193 out (160 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-nano:high__json_schema PASS 100% 95 in → 236 out (194 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4-nano:low PASS 100% 49 in → 72 out (38 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
openai/gpt-5.4-nano:low__json_schema PASS 100% 95 in → 42 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib-bare as submodule at lib\""]}
openai/gpt-5.4-nano:medium PASS 100% 49 in → 108 out (70 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
openai/gpt-5.4-nano:medium__json_schema PASS 100% 95 in → 95 out (52 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib as submodule\""]}
openai/gpt-5.4-nano:none PASS 100% 49 in → 36 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
openai/gpt-5.4-nano:none__json_schema PASS 100% 95 in → 38 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib as submodule\""]}
openai/gpt-5.4-nano:xhigh PASS 100% 49 in → 321 out (288 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 95 in → 1,193 out (1,151 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4:high PASS 100% 49 in → 188 out (155 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4:high__json_schema PASS 100% 95 in → 245 out (203 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4:low PASS 100% 49 in → 63 out (30 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4:low__json_schema PASS 100% 95 in → 74 out (32 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4:medium PASS 100% 49 in → 129 out (96 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4:medium__json_schema PASS 100% 95 in → 295 out (246 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.4:xhigh PASS 100% 49 in → 1,025 out (992 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.4:xhigh__json_schema PASS 100% 95 in → 565 out (516 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.5:high PASS 100% 49 in → 348 out (308 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.5:high__json_schema PASS 100% 95 in → 130 out (88 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.5:low PASS 100% 49 in → 73 out (40 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.5:low__json_schema PASS 100% 95 in → 84 out (42 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.5:medium PASS 100% 49 in → 80 out (47 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.5:medium__json_schema PASS 100% 95 in → 104 out (62 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.5:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.5:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.5:xhigh PASS 100% 49 in → 549 out (516 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.5:xhigh__json_schema PASS 100% 95 in → 565 out (516 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:high PASS 100% 49 in → 133 out (100 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:high__json_schema PASS 100% 95 in → 389 out (347 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:low PASS 100% 49 in → 82 out (49 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:low__json_schema PASS 100% 95 in → 80 out (38 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:max PASS 100% 49 in → 263 out (230 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:max__json_schema PASS 100% 95 in → 903 out (861 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:medium PASS 100% 49 in → 111 out (78 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:medium__json_schema PASS 100% 95 in → 124 out (82 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-luna:xhigh PASS 100% 49 in → 151 out (118 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 95 in → 558 out (516 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:high PASS 100% 49 in → 148 out (115 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:high__json_schema PASS 100% 95 in → 211 out (162 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:low PASS 100% 49 in → 175 out (142 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:low__json_schema PASS 100% 95 in → 153 out (111 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:max PASS 100% 49 in → 754 out (714 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:max__json_schema PASS 100% 95 in → 2,472 out (2,423 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:medium PASS 100% 49 in → 160 out (120 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:medium__json_schema PASS 100% 95 in → 182 out (133 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-sol:xhigh PASS 100% 49 in → 333 out (293 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 95 in → 274 out (225 reasoning)
git -c protocol.file.allow=always submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git -c protocol.file.allow=always submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:high PASS 100% 49 in → 101 out (68 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:high__json_schema PASS 100% 95 in → 186 out (144 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:low PASS 100% 49 in → 109 out (76 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:low__json_schema PASS 100% 95 in → 85 out (43 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:max PASS 100% 49 in → 183 out (150 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:max__json_schema PASS 100% 95 in → 392 out (350 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:medium PASS 100% 49 in → 77 out (44 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:medium__json_schema PASS 100% 95 in → 194 out (152 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:none PASS 100% 49 in → 31 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:none__json_schema PASS 100% 95 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-5.6-terra:xhigh PASS 100% 49 in → 121 out (88 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 95 in → 258 out (216 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-oss-120b:high PASS 100% 97 in → 1,166 out (1,230 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
openai/gpt-oss-120b:high__json_schema PASS 100% 112 in → 676 out (715 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib submodule\""]}
openai/gpt-oss-120b:low PASS 100% 110 in → 195 out (163 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
openai/gpt-oss-120b:low__json_schema PASS 100% 110 in → 49 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule pointing to ../lib-bare"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule pointing to ../lib-bare\"" ] }
openai/gpt-oss-120b:medium PASS 100% 108 in → 201 out (169 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule in lib"
openai/gpt-oss-120b:medium__json_schema PASS 100% 110 in → 180 out (121 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
openai/gpt-oss-120b:xhigh PASS 100% 110 in → 1,240 out (1,173 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 110 in → 35 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .gitmodules lib","git commit -m \"Add lib-bare as submodule\""]}
openai/gpt-oss-20b:high PASS 100% 112 in → 474 out (499 reasoning)
git submodule add ../lib-bare lib git commit -am "Add lib submodule"
openai/gpt-oss-20b:high__json_schema PASS 100% 108 in → 541 out (550 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule."
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule.\""]}
openai/gpt-oss-20b:low PASS 100% 112 in → 102 out (71 reasoning)
git submodule add ../lib-bare lib git commit -am "Add lib submodule"
openai/gpt-oss-20b:low__json_schema PASS 100% 110 in → 187 out (125 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as submodule\"" ] }
openai/gpt-oss-20b:medium PASS 100% 115 in → 216 out (193 reasoning)
git submodule add ../lib-bare lib git add .gitmodules git add lib git commit -m "Add lib submodule"
openai/gpt-oss-20b:medium__json_schema PASS 100% 97 in → 428 out (427 reasoning)
git submodule add ../lib-bare lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add submodule lib\""]}
openai/gpt-oss-20b:xhigh PASS 100% 97 in → 683 out (717 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 110 in → 5,382 out (5,243 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib submodule\""] }
poolside/laguna-m.1:high PASS 100% 56 in → 798 out (772 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a git submodule"
poolside/laguna-m.1:high__json_schema PASS 100% 56 in → 1,595 out (1,556 reasoning)
git submodule add ../lib-bare lib git commit -m 'Add lib as a submodule'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m 'Add lib as a submodule'" ] }
poolside/laguna-m.1:low PASS 100% 56 in → 784 out (763 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare submodule"
poolside/laguna-m.1:low__json_schema PASS 100% 56 in → 612 out (575 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule in lib"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule in lib\""]}
poolside/laguna-m.1:medium PASS 100% 56 in → 797 out (770 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a git submodule at lib"
poolside/laguna-m.1:medium__json_schema PASS 100% 56 in → 515 out (469 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare submodule\"" ] }
poolside/laguna-m.1:none PASS 100% 56 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib-bare as a submodule at lib"
poolside/laguna-m.1:none__json_schema PASS 100% 56 in → 43 out (0 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
poolside/laguna-m.1:xhigh PASS 100% 56 in → 537 out (506 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule"
poolside/laguna-m.1:xhigh__json_schema PASS 100% 56 in → 1,181 out (1,142 reasoning)
git submodule add ../lib-bare lib git commit -m 'Add lib-bare as a git submodule at lib'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m 'Add lib-bare as a git submodule at lib'" ] }
poolside/laguna-xs-2.1:high PASS 100% 56 in → 528 out (497 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule at lib"
poolside/laguna-xs-2.1:high__json_schema PASS 100% 56 in → 507 out (461 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a submodule\"" ] }
poolside/laguna-xs-2.1:low PASS 100% 56 in → 886 out (864 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
poolside/laguna-xs-2.1:low__json_schema PASS 100% 56 in → 941 out (911 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare as a submodule\""]}
poolside/laguna-xs-2.1:medium PASS 100% 56 in → 530 out (505 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 56 in → 323 out (280 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
poolside/laguna-xs-2.1:none PASS 100% 56 in → 62 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib" </think>git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
poolside/laguna-xs-2.1:none__json_schema PASS 100% 56 in → 52 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m 'Add lib-bare as submodule at lib'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m 'Add lib-bare as submodule at lib'" ] }
poolside/laguna-xs-2.1:xhigh PASS 100% 56 in → 746 out (721 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 56 in → 397 out (364 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add lib .gitmodules", "git commit -m \"Add lib submodule\""]}
poolside/laguna-xs.2:high PASS 100% 93 in → 139 out (110 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
poolside/laguna-xs.2:high__json_schema PASS 100% 93 in → 160 out (111 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as submodule\"" ] }
poolside/laguna-xs.2:low PASS 100% 93 in → 159 out (133 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as submodule"
poolside/laguna-xs.2:low__json_schema PASS 100% 93 in → 360 out (325 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as submodule\""]}
poolside/laguna-xs.2:medium PASS 100% 93 in → 142 out (112 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a submodule"
poolside/laguna-xs.2:medium__json_schema PASS 100% 93 in → 145 out (96 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as submodule\"" ] }
poolside/laguna-xs.2:none PASS 100% 93 in → 28 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
poolside/laguna-xs.2:none__json_schema PASS 100% 93 in → 35 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib as a submodule\""]}
poolside/laguna-xs.2:xhigh PASS 100% 93 in → 121 out (92 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as submodule"
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 93 in → 138 out (107 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\""]}
qwen/qwen3.6-27b:high PASS 100% 53 in → 2,427 out (2,372 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-27b:high__json_schema PASS 100% 53 in → 1,744 out (1,713 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
qwen/qwen3.6-27b:low PASS 100% 53 in → 747 out (701 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-27b:low__json_schema PASS 100% 53 in → 2,790 out (2,542 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib .gitmodules", "git commit -m \"Add lib-bare submodule at lib\"" ] }
qwen/qwen3.6-27b:medium PASS 100% 53 in → 1,017 out (980 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-27b:medium__json_schema PASS 100% 53 in → 1,538 out (1,487 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
qwen/qwen3.6-27b:none PASS 100% 55 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
qwen/qwen3.6-27b:none__json_schema PASS 100% 55 in → 40 out (0 reasoning)
git submodule add ../lib-bare lib git commit -m 'Add lib-bare as submodule'
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m 'Add lib-bare as submodule'" ] }
qwen/qwen3.6-27b:xhigh PASS 100% 53 in → 1,582 out (1,554 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 55 in → 47 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib as a submodule\"" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 53 in → 1,603 out (1,548 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 53 in → 1,351 out (1,286 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as submodule\"" ] }
qwen/qwen3.6-35b-a3b:low PASS 100% 53 in → 1,564 out (1,551 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 53 in → 1,067 out (954 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule\"" ] }
qwen/qwen3.6-35b-a3b:medium PASS 100% 53 in → 1,433 out (1,406 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 53 in → 1,489 out (1,420 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
qwen/qwen3.6-35b-a3b:none PASS 100% 55 in → 27 out (0 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib-bare as a submodule"
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 55 in → 54 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a git submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare as a git submodule at lib\"" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 53 in → 1,824 out (1,745 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as submodule at lib"
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 53 in → 1,956 out (1,870 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
qwen/qwen3.6-flash:high PASS 100% 53 in → 1,376 out (1,353 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
qwen/qwen3.6-flash:low PASS 100% 53 in → 1,278 out (1,251 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.6-flash:medium PASS 100% 53 in → 1,662 out (1,634 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as submodule at lib"
qwen/qwen3.6-flash:none PASS 100% 55 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib-bare as a submodule"
qwen/qwen3.6-flash:xhigh PASS 100% 53 in → 1,666 out (1,639 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.7-flash:high PASS 100% 53 in → 1,751 out (1,728 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
qwen/qwen3.7-flash:low PASS 100% 53 in → 1,934 out (1,902 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
qwen/qwen3.7-flash:max PASS 100% 53 in → 1,475 out (1,448 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.7-flash:medium PASS 100% 53 in → 2,019 out (1,989 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
qwen/qwen3.7-flash:none PASS 100% 55 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as a git submodule"
qwen/qwen3.7-flash:xhigh PASS 100% 53 in → 1,894 out (1,867 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
qwen/qwen3.7-max:high PASS 100% 53 in → 1,134 out (1,105 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
qwen/qwen3.7-max:low PASS 100% 53 in → 489 out (462 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule lib"
qwen/qwen3.7-max:medium PASS 100% 53 in → 976 out (950 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule"
qwen/qwen3.7-max:none PASS 100% 55 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
qwen/qwen3.7-max:xhigh PASS 100% 53 in → 728 out (702 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule"
qwen/qwen3.7-plus:high PASS 100% 53 in → 1,349 out (1,320 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule"
qwen/qwen3.7-plus:low PASS 100% 53 in → 1,031 out (1,002 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule"
qwen/qwen3.7-plus:medium PASS 100% 53 in → 1,041 out (1,011 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
qwen/qwen3.7-plus:none PASS 100% 55 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare as submodule at lib"
qwen/qwen3.7-plus:xhigh PASS 100% 53 in → 1,088 out (1,061 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
tencent/hy3:high PASS 100% 54 in → 1,841 out (1,814 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
tencent/hy3:high__json_schema PASS 100% 54 in → 2,364 out (2,415 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
tencent/hy3:low PASS 100% 54 in → 429 out (402 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
tencent/hy3:low__json_schema PASS 100% 54 in → 743 out (668 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
tencent/hy3:medium PASS 100% 54 in → 628 out (601 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
tencent/hy3:medium__json_schema PASS 100% 54 in → 713 out (660 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
tencent/hy3:none PASS 100% 57 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
tencent/hy3:none__json_schema PASS 100% 57 in → 43 out (0 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
tencent/hy3:xhigh PASS 100% 54 in → 1,753 out (1,698 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
tencent/hy3:xhigh__json_schema PASS 100% 54 in → 1,379 out (1,286 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
thinkingmachines/inkling-small:high PASS 100% 58 in → 374 out (346 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
thinkingmachines/inkling-small:high__json_schema PASS 100% 58 in → 572 out (526 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
thinkingmachines/inkling-small:low PASS 100% 58 in → 72 out (40 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
thinkingmachines/inkling-small:low__json_schema PASS 100% 58 in → 81 out (32 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
thinkingmachines/inkling-small:max PASS 100% 58 in → 194 out (161 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
thinkingmachines/inkling-small:max__json_schema PASS 100% 58 in → 377 out (331 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib-bare submodule\"" ] }
thinkingmachines/inkling-small:medium PASS 100% 58 in → 155 out (122 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
thinkingmachines/inkling-small:medium__json_schema PASS 100% 58 in → 235 out (186 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
thinkingmachines/inkling-small:none PASS 100% 56 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
thinkingmachines/inkling-small:xhigh PASS 100% 58 in → 212 out (178 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib-bare submodule"
thinkingmachines/inkling:high PASS 100% 58 in → 297 out (268 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
thinkingmachines/inkling:low PASS 100% 58 in → 63 out (33 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
thinkingmachines/inkling:max PASS 100% 58 in → 271 out (245 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
thinkingmachines/inkling:max__json_schema PASS 100% 58 in → 209 out (175 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
thinkingmachines/inkling:medium PASS 100% 58 in → 94 out (64 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
thinkingmachines/inkling:medium__json_schema PASS 100% 58 in → 86 out (42 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add submodule\"" ] }
thinkingmachines/inkling:none__json_schema PASS 100% 56 in → 51 out (0 reasoning)
git submodule add ../lib-bare lib git commit -m 'Add lib submodule'
JSON Schema Structured Output
(raw) { "commands" : [ "git submodule add ../lib-bare lib" , "git commit -m 'Add lib submodule'" ] }
thinkingmachines/inkling:xhigh PASS 100% 58 in → 264 out (234 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
thinkingmachines/inkling:xhigh__json_schema PASS 100% 58 in → 499 out (454 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.3:high PASS 100% 227 in → 1,443 out (1,422 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
x-ai/grok-4.3:high__json_schema PASS 100% 293 in → 579 out (539 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.3:low PASS 100% 233 in → 292 out (267 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.3:low__json_schema PASS 100% 299 in → 407 out (367 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.3:max PASS 100% 227 in → 1,441 out (1,416 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.3:max__json_schema PASS 100% 293 in → 1,378 out (1,338 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.3:medium PASS 100% 233 in → 1,023 out (948 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule" The first command registers the submodule (creating `.gitmodules` and the `lib` gitlink). The subsequent `git add` and `commit` make the requested changes persistent; the commit message is a minimal descriptive default because none was supplied.
x-ai/grok-4.3:medium__json_schema PASS 100% 299 in → 641 out (601 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add submodule lib\"" ] }
x-ai/grok-4.3:none PASS 100% 225 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare as submodule at lib"
x-ai/grok-4.3:none__json_schema PASS 100% 291 in → 26 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add submodule lib"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add submodule lib\""]}
x-ai/grok-4.3:xhigh PASS 100% 227 in → 562 out (537 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.3:xhigh__json_schema PASS 100% 293 in → 1,269 out (1,229 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.5:high PASS 100% 248 in → 530 out (509 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
x-ai/grok-4.5:high__json_schema PASS 100% 327 in → 613 out (574 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare as submodule at lib\"" ] }
x-ai/grok-4.5:low PASS 100% 248 in → 177 out (153 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.5:low__json_schema PASS 100% 327 in → 345 out (302 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.5:max PASS 100% 248 in → 557 out (539 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
x-ai/grok-4.5:max__json_schema PASS 100% 327 in → 619 out (576 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\"" ] }
x-ai/grok-4.5:medium PASS 100% 248 in → 401 out (377 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.5:medium__json_schema PASS 100% 327 in → 436 out (412 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
x-ai/grok-4.5:xhigh PASS 100% 248 in → 468 out (444 reasoning)
git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
x-ai/grok-4.5:xhigh__json_schema PASS 100% 327 in → 473 out (445 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
z-ai/glm-4.7-flash:high PASS 100% 47 in → 422 out (395 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare submodule"
z-ai/glm-4.7-flash:low PASS 100% 47 in → 4,509 out (4,476 reasoning)
git submodule add ../lib-bare lib git add .gitmodules .git/config git add lib/ git commit -m "Add lib submodule"
z-ai/glm-4.7-flash:medium PASS 100% 47 in → 493 out (532 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare submodule"
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 47 in → 502 out (450 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare as a submodule\""]}
z-ai/glm-4.7-flash:none PASS 100% 47 in → 27 out (0 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule at path 'lib'"
z-ai/glm-4.7-flash:none__json_schema PASS 100% 47 in → 38 out (0 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare submodule at path lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare submodule at path lib\"" ] }
z-ai/glm-4.7-flash:xhigh PASS 100% 47 in → 387 out (351 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare as a submodule"
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 47 in → 742 out (712 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) {"commands": ["git submodule add ../lib-bare lib", "git commit -m \"Add lib-bare submodule\""]}
z-ai/glm-5.2:high PASS 100% 54 in → 107 out (82 reasoning)
git submodule add ../lib-bare lib git add lib git commit -m "Add lib submodule"
z-ai/glm-5.2:high__json_schema PASS 100% 239 in → 188 out (164 reasoning)
git submodule add ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git commit -m \"Add lib submodule\""]}
z-ai/glm-5.2:low PASS 100% 54 in → 407 out (362 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib submodule"
z-ai/glm-5.2:low__json_schema PASS 100% 54 in → 312 out (268 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib .gitmodules", "git commit -m \"Add lib-bare submodule\"" ] }
z-ai/glm-5.2:medium PASS 100% 54 in → 500 out (473 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib submodule"
z-ai/glm-5.2:none PASS 100% 48 in → 35 out (0 reasoning)
```bash git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare as submodule at lib" ```
z-ai/glm-5.2:none__json_schema PASS 100% 48 in → 45 out (0 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib-bare as a submodule at lib"
JSON Schema Structured Output
(raw) { "commands": [ "git submodule add ../lib-bare lib", "git add lib .gitmodules", "git commit -m \"Add lib-bare as a submodule at lib\"" ] }
z-ai/glm-5.2:xhigh PASS 100% 47 in → 567 out (517 reasoning)
git submodule add ../lib-bare lib git add lib .gitmodules git commit -m "Add lib submodule"
z-ai/glm-5.2:xhigh__json_schema PASS 100% 239 in → 104 out (76 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) {"commands":["git submodule add ../lib-bare lib","git add .","git commit -m \"Add lib submodule\""]}
minimax/minimax-m3:low FAIL 66.7% 218 in → 67 out (45 reasoning)
git submodule add ../lib-bare lib git add .gitmoduleslib git commit -m "Add lib submodule"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}]
thinkingmachines/inkling:none FAIL 66.7% 58 in → 30 out (0 reasoning)
git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule at lib"<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}]
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 52 in → 371 out (397 reasoning)
git submodule add../lib-bare lib git commit -m "Add lib-bare as a submodule"
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
arcee-ai/trinity-mini:medium FAIL 0% 52 in → 502 out (561 reasoning)
git submodule add ../lib-bare libgit commit -m "Add lib submodule"
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 52 in → 362 out (389 reasoning)
git submoduleadd ../lib-bare lib git commit -m "Add lib-bare as a submodule"
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 52 in → 372 out (375 reasoning)
git submoduleadd ../lib-bare lib git commit -m "Add lib submodule"
JSON Schema Structured Output
(raw) { "commands": [ "git submoduleadd ../lib-bare lib", "git commit -m \"Add lib submodule\"" ] }
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
deepseek/deepseek-v4-flash:none__json_schema FAIL 0%
Invalid JSON. Output: ```bash git submodule add ../lib-bare lib git add lib git commit -m "Add lib-bare as submodule at lib" ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
ibm-granite/granite-4.1-8b FAIL 0% 52 in → 27 out
```bash git submodule add ../lib-raw lib git add lib git commit -m "Add lib submodule" ```
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git commit -m "Added git submodule for lib-bare at path lib"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:medium FAIL 0% 85 in → 522 out (551 reasoning)
submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib as a submodule"
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
minimax/minimax-m3:medium__json_schema FAIL 0% 205 in → 60 out (0 reasoning)
cd ../lib-bare && git submodule add ../lib-bare lib cd ../lib-bare && git add .gitmodules lib && git commit -m "Add lib-bare as submodule at path 'lib'"
JSON Schema Structured Output
(raw) { "commands": [ "cd ../lib-bare && git submodule add ../lib-bare lib", "cd ../lib-bare && git add .gitmodules lib && git commit -m \"Add lib-bare as submodule at path 'lib'\"" ] }
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
moonshotai/kimi-k2.6:high__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:medium FAIL 0% 59 in → 243 out (236 reasoning)
git submoduleadd ../lib-bare lib git add lib git commit -m "Add lib submodule"
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8a87be56-e694-975d-9052-89e2cdf111a3","request_id":"8a87be56-e694-975d-9052-89e2cdf111a3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e709119d-f8dc-945b-9541-52df7fb005f5","request_id":"e709119d-f8dc-945b-9541-52df7fb005f5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c5666f35-1300-9587-acfa-93fa59fa8a9d","request_id":"c5666f35-1300-9587-acfa-93fa59fa8a9d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-394f97b3-a858-9d76-8d77-5df0eef61aee","request_id":"394f97b3-a858-9d76-8d77-5df0eef61aee"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b48f8cff-0ea7-9760-8072-5373434d2290","request_id":"b48f8cff-0ea7-9760-8072-5373434d2290"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b6a06187-cfe1-9770-9b0f-439d887b86cd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-12a68ffb-b472-97ad-b885-354fc50bc803"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-94e4f4c0-bfc6-9074-80bd-dc77f026a388"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b7f7ce08-52f4-9300-bedb-12a213e6a0be"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8a7f4587-e149-910f-8bda-44f46a3bb3c4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7aa4fd32-4d0b-929d-bfde-d5fdc7041679"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3c64e0fc-60b9-9622-8575-e4c49125b279","request_id":"3c64e0fc-60b9-9622-8575-e4c49125b279"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-38774d23-0de0-9cac-b278-53b7b785326c","request_id":"38774d23-0de0-9cac-b278-53b7b785326c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0ac52915-efa8-9a27-a1f5-a24cba8ba8a6","request_id":"0ac52915-efa8-9a27-a1f5-a24cba8ba8a6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7c0134f1-946b-9dd6-b558-8ede2498f048","request_id":"7c0134f1-946b-9dd6-b558-8ede2498f048"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7f2187cd-fa39-9cf9-970c-9331c03fc6ff","request_id":"7f2187cd-fa39-9cf9-970c-9331c03fc6ff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3b25235d-788d-997d-8431-8d5255ed0363","request_id":"3b25235d-788d-997d-8431-8d5255ed0363"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a87c2375-10f6-9c69-9994-f4b51fb23b28","request_id":"a87c2375-10f6-9c69-9994-f4b51fb23b28"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-055b7eb3-6d31-9dd7-b413-15d20dbaebbf","request_id":"055b7eb3-6d31-9dd7-b413-15d20dbaebbf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5f678092-a3d3-913f-a7ff-7484702ca9e7","request_id":"5f678092-a3d3-913f-a7ff-7484702ca9e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1062da8e-aabf-9b1a-8d14-b18e44b863af","request_id":"1062da8e-aabf-9b1a-8d14-b18e44b863af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git add .gitmodules lib git commit -m "Add lib submodule"{"commands": ["git submodule add ../lib-bare lib", "git add .gitmodules lib", "git commit -m \"Add lib submodule\""]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: json_schema response format is not supported for model: thinkingmachines/Inkling-Small
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git add . git commit -m "Add lib-bare submodule at lib"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: git submodule add ../lib-bare lib git add . git commit -m "Add lib submodule"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:medium__json_schema FAIL 0% 56 in → 350 out (292 reasoning)
In repository `..` add `../lib-bare` as a submodule at `lib` After submodule add, stage changes in `..` After submodule add, commit changes in `..`
JSON Schema Structured Output
(raw) { "commands": [ "In repository `..` add `../lib-bare` as a submodule at `lib`", "After submodule add, stage changes in `..`", "After submodule add, commit changes in `..`" ] }
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git ls-tree HEAD .gitmodules', 'contains': '.gitmodules'}, {'type': 'git_output', 'command': 'git ls-tree HEAD lib', 'contains': '160000 commit'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.path', 'exact': 'lib'}, {'type': 'git_output', 'command': 'git config -f .gitmodules --get submodule.lib.url', 'exact': '../lib-bare'}]