Add a submodule
Tests ability to add a git submodule to a repository. Evaluates basic submodule-add command knowledge.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
Prompt
Add the repository at '../lib-bare' as a git submodule at path 'lib'. Output ONLY the git command to run, nothing else.
Expected
git submodule add ../lib-bare lib

Scoped model quality, cost, API time, and token usage for submodule_usage/f001.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 61 in → 39 out (31 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-fable-5:high__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-fable-5:low PASS 100% 61 in → 39 out (31 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-fable-5:low__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-fable-5:max PASS 100% 61 in → 96 out (27 reasoning)
git submodule add ../lib-bare lib
anthropic/claude-fable-5:max__json_schema PASS 100% 271 in → 95 out (41 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-fable-5:medium PASS 100% 61 in → 45 out (28 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-fable-5:xhigh PASS 100% 61 in → 60 out (32 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-haiku-4.5:high PASS 100% 77 in → 130 out (105 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 240 in → 159 out (129 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-haiku-4.5:low PASS 100% 77 in → 154 out (114 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 240 in → 133 out (101 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-haiku-4.5:medium PASS 100% 77 in → 134 out (100 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 240 in → 143 out (109 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-haiku-4.5:none PASS 100% 48 in → 17 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 210 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 77 in → 140 out (108 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 240 in → 163 out (137 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:high PASS 100% 48 in → 58 out (27 reasoning)
git submodule add ../lib-bare lib
anthropic/claude-opus-4.6:high__json_schema PASS 100% 211 in → 59 out (23 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:low PASS 100% 48 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
anthropic/claude-opus-4.6:low__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:max PASS 100% 48 in → 58 out (27 reasoning)
git submodule add ../lib-bare lib
anthropic/claude-opus-4.6:max__json_schema PASS 100% 211 in → 56 out (19 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:medium PASS 100% 48 in → 18 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:none PASS 100% 48 in → 17 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.6:xhigh PASS 100% 48 in → 18 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 211 in → 48 out (15 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:high PASS 100% 66 in → 23 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.7:high__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:low PASS 100% 66 in → 20 out (0 reasoning)
`git submodule add ../lib-bare lib`
anthropic/claude-opus-4.7:low__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:max PASS 100% 66 in → 23 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:medium PASS 100% 66 in → 20 out (0 reasoning)
`git submodule add ../lib-bare lib`
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:none PASS 100% 66 in → 23 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.7:xhigh PASS 100% 66 in → 23 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 276 in → 25 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:high PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:low PASS 100% 61 in → 16 out (0 reasoning)
git submodule add ../lib-bare lib
anthropic/claude-opus-4.8:low__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:max PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:medium PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:none PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 271 in → 22 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-opus-4.8:xhigh PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:high PASS 100% 48 in → 54 out (18 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:low PASS 100% 48 in → 17 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:max PASS 100% 48 in → 95 out (60 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 211 in → 83 out (46 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:medium PASS 100% 48 in → 17 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:none PASS 100% 48 in → 17 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 48 in → 54 out (18 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 211 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-5:high PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-5:low PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-5:medium PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-5:none PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 271 in → 22 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
anthropic/claude-sonnet-5:xhigh PASS 100% 61 in → 20 out (0 reasoning)
``` git submodule add ../lib-bare lib ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 271 in → 23 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
arcee-ai/trinity-large-thinking:high PASS 100% 45 in → 153 out (142 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-large-thinking:low PASS 100% 45 in → 184 out (173 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-large-thinking:medium PASS 100% 45 in → 199 out (188 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-large-thinking:xhigh PASS 100% 45 in → 195 out (184 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-mini:high PASS 100% 45 in → 292 out (318 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-mini:high__json_schema PASS 100% 45 in → 283 out (299 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command":"git submodule add ../lib-bare lib" }
arcee-ai/trinity-mini:low__json_schema PASS 100% 45 in → 407 out (457 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
arcee-ai/trinity-mini:medium PASS 100% 45 in → 310 out (342 reasoning)
git submodule add ../lib-bare lib
arcee-ai/trinity-mini:medium__json_schema PASS 100% 45 in → 332 out (356 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 45 in → 327 out (362 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 40 in → 81 out (70 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 119 in → 122 out (105 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 40 in → 93 out (83 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 40 in → 119 out (100 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 119 in → 74 out (61 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:medium PASS 100% 40 in → 73 out (67 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 40 in → 131 out (115 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 40 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 40 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 119 in → 83 out (71 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 40 in → 113 out (99 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
deepseek/deepseek-v4-flash:high PASS 100% 40 in → 53 out (42 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 42 in → 78 out (59 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash:low PASS 100% 40 in → 63 out (52 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 42 in → 101 out (83 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash:medium PASS 100% 40 in → 106 out (96 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 40 in → 89 out (70 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash:none PASS 100% 40 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 40 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 40 in → 105 out (93 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 119 in → 146 out (134 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-pro:high PASS 100% 40 in → 113 out (102 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 40 in → 70 out (55 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-pro:low PASS 100% 40 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 227 in → 245 out (229 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
deepseek/deepseek-v4-pro:medium PASS 100% 40 in → 113 out (102 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 42 in → 125 out (107 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-pro:none PASS 100% 40 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 40 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
deepseek/deepseek-v4-pro:xhigh PASS 100% 119 in → 324 out (313 reasoning)
git submodule add ../lib-bare lib
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 40 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemini-3-flash-preview:high PASS 100% 36 in → 108 out (100 reasoning)
git submodule add ../lib-bare lib
google/gemini-3-flash-preview:high__json_schema PASS 100% 93 in → 242 out (229 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3-flash-preview:low PASS 100% 36 in → 153 out (145 reasoning)
git submodule add ../lib-bare lib
google/gemini-3-flash-preview:low__json_schema PASS 100% 93 in → 211 out (199 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3-flash-preview:medium PASS 100% 36 in → 131 out (123 reasoning)
git submodule add ../lib-bare lib
google/gemini-3-flash-preview:medium__json_schema PASS 100% 93 in → 192 out (180 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3-flash-preview:xhigh PASS 100% 36 in → 124 out (116 reasoning)
git submodule add ../lib-bare lib
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 37 in → 76 out (64 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 37 in → 217 out (209 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 37 in → 412 out (399 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 37 in → 135 out (127 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 37 in → 146 out (129 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 36 in → 88 out (80 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 37 in → 181 out (163 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 37 in → 206 out (198 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 93 in → 171 out (158 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.1-pro-preview:high PASS 100% 36 in → 157 out (149 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 93 in → 260 out (248 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.1-pro-preview:low PASS 100% 36 in → 198 out (188 reasoning)
`git submodule add ../lib-bare lib`
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 93 in → 141 out (128 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.1-pro-preview:medium PASS 100% 36 in → 221 out (213 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 93 in → 299 out (286 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 36 in → 202 out (194 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 93 in → 285 out (273 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.5-flash-lite:high PASS 100% 36 in → 239 out (231 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 93 in → 342 out (330 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.5-flash-lite:low PASS 100% 37 in → 10 out (0 reasoning)
`git submodule add ../lib-bare lib`
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 93 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemini-3.5-flash-lite:max PASS 100% 36 in → 145 out (137 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 93 in → 283 out (270 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.5-flash-lite:medium PASS 100% 37 in → 153 out (145 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 37 in → 245 out (232 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 37 in → 189 out (181 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 93 in → 426 out (413 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.5-flash:high PASS 100% 36 in → 194 out (186 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash:high__json_schema PASS 100% 93 in → 266 out (253 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.5-flash:low PASS 100% 36 in → 131 out (123 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash:low__json_schema PASS 100% 93 in → 74 out (61 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.5-flash:medium PASS 100% 36 in → 293 out (285 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash:medium__json_schema PASS 100% 93 in → 247 out (235 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.5-flash:xhigh PASS 100% 36 in → 210 out (202 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 93 in → 275 out (262 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemini-3.6-flash:high PASS 100% 36 in → 224 out (216 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.6-flash:high__json_schema PASS 100% 37 in → 220 out (208 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.6-flash:low PASS 100% 37 in → 135 out (127 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.6-flash:low__json_schema PASS 100% 37 in → 12 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.6-flash:max PASS 100% 36 in → 293 out (285 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.6-flash:max__json_schema PASS 100% 37 in → 337 out (325 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.6-flash:medium PASS 100% 37 in → 235 out (227 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.6-flash:medium__json_schema PASS 100% 37 in → 188 out (176 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemini-3.6-flash:xhigh PASS 100% 37 in → 229 out (221 reasoning)
git submodule add ../lib-bare lib
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 37 in → 289 out (277 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
google/gemma-4-26b-a4b-it:high PASS 100% 52 in → 131 out (105 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 51 in → 144 out (108 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-26b-a4b-it:low PASS 100% 52 in → 135 out (119 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 52 in → 163 out (132 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-26b-a4b-it:medium PASS 100% 52 in → 132 out (107 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 52 in → 146 out (113 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-26b-a4b-it:none PASS 100% 49 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 49 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 52 in → 125 out (103 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 52 in → 129 out (96 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-31b-it:high PASS 100% 52 in → 117 out (95 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-31b-it:high__json_schema PASS 100% 51 in → 120 out (93 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemma-4-31b-it:low PASS 100% 52 in → 114 out (90 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-31b-it:low__json_schema PASS 100% 51 in → 119 out (93 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-31b-it:medium PASS 100% 51 in → 113 out (83 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-31b-it:medium__json_schema PASS 100% 51 in → 129 out (93 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
google/gemma-4-31b-it:none PASS 100% 49 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-31b-it:none__json_schema PASS 100% 49 in → 14 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
google/gemma-4-31b-it:xhigh PASS 100% 52 in → 114 out (90 reasoning)
git submodule add ../lib-bare lib
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 52 in → 136 out (102 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
ibm-granite/granite-4.1-8b PASS 100% 45 in → 9 out
git submodule add ../lib-bare lib
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
inclusionai/ling-2.6-flash PASS 100% 56 in → 10 out
git submodule add ../lib-bare lib
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.5:high__json_schema PASS 100% 73 in → 86 out (70 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.5:low PASS 100% 73 in → 226 out (239 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.5:low__json_schema PASS 100% 76 in → 113 out (96 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.5:medium PASS 100% 73 in → 94 out (85 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.5:medium__json_schema PASS 100% 73 in → 75 out (57 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.5:xhigh PASS 100% 73 in → 89 out (82 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.7:high PASS 100% 72 in → 186 out (208 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.7:high__json_schema PASS 100% 72 in → 97 out (95 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.7:low PASS 100% 76 in → 245 out (236 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.7:low__json_schema PASS 100% 205 in → 280 out (262 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m2.7:medium PASS 100% 73 in → 116 out (117 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.7:medium__json_schema PASS 100% 205 in → 232 out (218 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
minimax/minimax-m2.7:xhigh PASS 100% 73 in → 160 out (150 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 205 in → 258 out (244 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
minimax/minimax-m3:high PASS 100% 211 in → 44 out (37 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m3:high__json_schema PASS 100% 159 in → 16 out (15 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m3:low PASS 100% 211 in → 66 out (62 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m3:low__json_schema PASS 100% 159 in → 14 out (13 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
minimax/minimax-m3:medium PASS 100% 208 in → 67 out (66 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m3:medium__json_schema PASS 100% 159 in → 16 out (15 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
minimax/minimax-m3:xhigh PASS 100% 211 in → 47 out (45 reasoning)
git submodule add ../lib-bare lib
minimax/minimax-m3:xhigh__json_schema PASS 100% 159 in → 17 out (16 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
mistralai/devstral-2512 PASS 100% 39 in → 15 out
```bash git submodule add ../lib-bare lib ```
mistralai/devstral-2512__json_schema PASS 100% 39 in → 15 out
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
mistralai/mistral-medium-3-5:high PASS 100% 51 in → 219 out (228 reasoning)
git submodule add ../lib-bare lib
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 39 in → 232 out (227 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
mistralai/mistral-medium-3-5:low PASS 100% 51 in → 344 out (360 reasoning)
git submodule add ../lib-bare lib
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 39 in → 181 out (164 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
mistralai/mistral-medium-3-5:medium PASS 100% 51 in → 276 out (267 reasoning)
git submodule add ../lib-bare lib
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 39 in → 160 out (152 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
mistralai/mistral-medium-3-5:none PASS 100% 51 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 39 in → 15 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 51 in → 402 out (427 reasoning)
git submodule add ../lib-bare lib
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 39 in → 350 out (338 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
moonshotai/kimi-k2.6:high PASS 100% 42 in → 157 out (147 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.6:high__json_schema PASS 100% 43 in → 196 out (206 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
moonshotai/kimi-k2.6:low PASS 100% 43 in → 109 out (99 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.6:low__json_schema PASS 100% 42 in → 548 out (160 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib"}
moonshotai/kimi-k2.6:medium PASS 100% 42 in → 141 out (131 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.6:xhigh PASS 100% 43 in → 142 out (132 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 43 in → 191 out (201 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
moonshotai/kimi-k2.7-code:high PASS 100% 43 in → 86 out (78 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 43 in → 50 out (32 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
moonshotai/kimi-k2.7-code:low PASS 100% 43 in → 101 out (91 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 43 in → 87 out (69 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
moonshotai/kimi-k2.7-code:medium PASS 100% 43 in → 70 out (60 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 43 in → 126 out (111 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 43 in → 124 out (114 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 43 in → 204 out (189 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
moonshotai/kimi-k3:high PASS 100% 120 in → 76 out (53 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:high__json_schema PASS 100% 120 in → 134 out (111 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:low PASS 100% 120 in → 80 out (57 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:low__json_schema PASS 100% 120 in → 90 out (67 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:max PASS 100% 120 in → 96 out (70 reasoning)
``` git submodule add ../lib-bare lib ```
moonshotai/kimi-k3:max__json_schema PASS 100% 120 in → 94 out (66 reasoning)
```bash git submodule add ../lib-bare lib ```
moonshotai/kimi-k3:medium PASS 100% 120 in → 172 out (149 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:medium__json_schema PASS 100% 120 in → 81 out (58 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:xhigh PASS 100% 120 in → 86 out (63 reasoning)
git submodule add ../lib-bare lib
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 120 in → 76 out (53 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 52 in → 309 out (321 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 52 in → 3,142 out (3,517 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 52 in → 124 out (123 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 52 in → 91 out (74 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 52 in → 154 out (139 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 52 in → 149 out (146 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 52 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 52 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 52 in → 1,785 out (1,995 reasoning)
git submodule add '../lib-bare' lib
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 52 in → 103 out (88 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 52 in → 64 out (50 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 52 in → 73 out (51 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 52 in → 95 out (94 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 52 in → 61 out (40 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 52 in → 80 out (74 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 52 in → 58 out (38 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 52 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 52 in → 18 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 52 in → 75 out (62 reasoning)
git submodule add ../lib-bare lib
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 52 in → 95 out (74 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
openai/gpt-5.4-mini:high PASS 100% 42 in → 73 out (58 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-mini:high__json_schema PASS 100% 77 in → 164 out (141 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-mini:low PASS 100% 42 in → 36 out (21 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-mini:low__json_schema PASS 100% 77 in → 57 out (34 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-mini:medium PASS 100% 42 in → 57 out (42 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-mini:medium__json_schema PASS 100% 77 in → 73 out (50 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-mini:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-mini:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-mini:xhigh PASS 100% 42 in → 109 out (94 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 77 in → 187 out (164 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-nano:high PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-nano:high__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-nano:low PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-nano:low__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-nano:medium PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-nano:medium__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-nano:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-nano:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4-nano:xhigh PASS 100% 42 in → 97 out (82 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 77 in → 535 out (512 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4:high PASS 100% 42 in → 111 out (96 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4:high__json_schema PASS 100% 77 in → 90 out (67 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4:low PASS 100% 42 in → 50 out (35 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4:low__json_schema PASS 100% 77 in → 59 out (36 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4:medium PASS 100% 42 in → 75 out (60 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4:medium__json_schema PASS 100% 77 in → 65 out (42 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.4:xhigh PASS 100% 42 in → 163 out (148 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.4:xhigh__json_schema PASS 100% 77 in → 156 out (133 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.5:high PASS 100% 42 in → 56 out (41 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.5:high__json_schema PASS 100% 77 in → 67 out (44 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.5:low PASS 100% 42 in → 47 out (32 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.5:low__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.5:medium PASS 100% 42 in → 45 out (30 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.5:medium__json_schema PASS 100% 77 in → 84 out (60 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
openai/gpt-5.5:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.5:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.5:xhigh PASS 100% 42 in → 98 out (83 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.5:xhigh__json_schema PASS 100% 77 in → 539 out (516 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:high PASS 100% 42 in → 49 out (34 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:high__json_schema PASS 100% 77 in → 51 out (28 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:low PASS 100% 42 in → 46 out (31 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:low__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:max PASS 100% 42 in → 59 out (44 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:max__json_schema PASS 100% 77 in → 65 out (42 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:medium PASS 100% 42 in → 49 out (34 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:medium__json_schema PASS 100% 77 in → 46 out (23 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-luna:xhigh PASS 100% 42 in → 70 out (55 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 77 in → 65 out (42 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:high PASS 100% 42 in → 43 out (28 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:high__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:low PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:low__json_schema PASS 100% 77 in → 45 out (22 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:max PASS 100% 42 in → 77 out (62 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:max__json_schema PASS 100% 77 in → 377 out (354 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:medium PASS 100% 42 in → 44 out (29 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:medium__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-sol:xhigh PASS 100% 42 in → 139 out (124 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 77 in → 222 out (199 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:high PASS 100% 42 in → 46 out (31 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:high__json_schema PASS 100% 77 in → 72 out (49 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:low PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:low__json_schema PASS 100% 77 in → 128 out (105 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:max PASS 100% 42 in → 81 out (66 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:max__json_schema PASS 100% 77 in → 98 out (75 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:medium PASS 100% 42 in → 35 out (20 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:medium__json_schema PASS 100% 77 in → 79 out (56 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:none PASS 100% 42 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:none__json_schema PASS 100% 77 in → 21 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-5.6-terra:xhigh PASS 100% 42 in → 45 out (30 reasoning)
git submodule add ../lib-bare lib
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 77 in → 116 out (93 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-120b:high PASS 100% 105 in → 199 out (190 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-120b:high__json_schema PASS 100% 101 in → 256 out (272 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-120b:low PASS 100% 107 in → 49 out (29 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-120b:low__json_schema PASS 100% 103 in → 17 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
openai/gpt-oss-120b:medium PASS 100% 101 in → 85 out (67 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-120b:medium__json_schema PASS 100% 90 in → 152 out (130 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
openai/gpt-oss-120b:xhigh PASS 100% 103 in → 135 out (112 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 101 in → 177 out (168 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-20b:high PASS 100% 90 in → 212 out (211 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-20b:high__json_schema PASS 100% 103 in → 177 out (147 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-20b:low PASS 100% 90 in → 33 out (13 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-20b:low__json_schema PASS 100% 103 in → 52 out (28 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-20b:medium PASS 100% 101 in → 72 out (54 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-20b:medium__json_schema PASS 100% 105 in → 152 out (132 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
openai/gpt-oss-20b:xhigh PASS 100% 101 in → 267 out (281 reasoning)
git submodule add ../lib-bare lib
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 105 in → 155 out (136 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-m.1:high PASS 100% 49 in → 233 out (223 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-m.1:high__json_schema PASS 100% 49 in → 238 out (223 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-m.1:low PASS 100% 49 in → 253 out (243 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-m.1:low__json_schema PASS 100% 49 in → 267 out (253 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-m.1:medium PASS 100% 49 in → 304 out (295 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-m.1:medium__json_schema PASS 100% 49 in → 322 out (308 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-m.1:none PASS 100% 49 in → 15 out (0 reasoning)
```bash git submodule add ../lib-bare lib ```
poolside/laguna-m.1:none__json_schema PASS 100% 49 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-m.1:xhigh PASS 100% 49 in → 324 out (313 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-m.1:xhigh__json_schema PASS 100% 49 in → 158 out (139 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-xs-2.1:high PASS 100% 49 in → 362 out (354 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs-2.1:high__json_schema PASS 100% 49 in → 240 out (221 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-xs-2.1:low PASS 100% 49 in → 282 out (274 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs-2.1:low__json_schema PASS 100% 49 in → 209 out (195 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs-2.1:medium PASS 100% 49 in → 224 out (216 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 49 in → 295 out (276 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-xs-2.1:none PASS 100% 49 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs-2.1:none__json_schema PASS 100% 49 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-xs-2.1:xhigh PASS 100% 49 in → 271 out (263 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 49 in → 324 out (310 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs.2:high PASS 100% 86 in → 78 out (68 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:high__json_schema PASS 100% 86 in → 102 out (88 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs.2:low PASS 100% 86 in → 78 out (68 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:low__json_schema PASS 100% 86 in → 91 out (77 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs.2:medium PASS 100% 86 in → 105 out (95 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:medium__json_schema PASS 100% 86 in → 62 out (43 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
poolside/laguna-xs.2:none PASS 100% 86 in → 10 out (0 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:none__json_schema PASS 100% 86 in → 14 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
poolside/laguna-xs.2:xhigh PASS 100% 86 in → 47 out (36 reasoning)
git submodule add ../lib-bare lib
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 86 in → 82 out (63 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-27b:high PASS 100% 46 in → 767 out (1 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-27b:high__json_schema PASS 100% 46 in → 824 out (820 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-27b:low PASS 100% 46 in → 277 out (255 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-27b:medium PASS 100% 46 in → 291 out (278 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-27b:none PASS 100% 48 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-27b:none__json_schema PASS 100% 48 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-27b:xhigh PASS 100% 46 in → 681 out (668 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 46 in → 958 out (974 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-35b-a3b:high PASS 100% 46 in → 605 out (576 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 46 in → 327 out (303 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-35b-a3b:low PASS 100% 46 in → 298 out (280 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 46 in → 415 out (399 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 46 in → 864 out (854 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 46 in → 260 out (241 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-35b-a3b:none PASS 100% 48 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 48 in → 19 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 46 in → 737 out (743 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 46 in → 296 out (280 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
qwen/qwen3.6-flash:high PASS 100% 46 in → 708 out (695 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-flash:low PASS 100% 46 in → 262 out (249 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-flash:medium PASS 100% 46 in → 498 out (485 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-flash:none PASS 100% 48 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.6-flash:xhigh PASS 100% 46 in → 522 out (509 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:high PASS 100% 46 in → 395 out (382 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:low PASS 100% 46 in → 612 out (599 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:max PASS 100% 46 in → 461 out (448 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:medium PASS 100% 46 in → 732 out (719 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:none PASS 100% 48 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-flash:xhigh PASS 100% 46 in → 657 out (644 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-max:high PASS 100% 46 in → 295 out (282 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-max:low PASS 100% 46 in → 464 out (451 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-max:medium PASS 100% 46 in → 487 out (474 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-max:none PASS 100% 48 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-max:xhigh PASS 100% 46 in → 497 out (484 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-plus:high PASS 100% 46 in → 364 out (351 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-plus:low PASS 100% 46 in → 499 out (486 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-plus:medium PASS 100% 46 in → 277 out (264 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-plus:none PASS 100% 48 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
qwen/qwen3.7-plus:xhigh PASS 100% 46 in → 259 out (246 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:high PASS 100% 47 in → 260 out (250 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:high__json_schema PASS 100% 47 in → 144 out (127 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
tencent/hy3:low PASS 100% 47 in → 89 out (79 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:low__json_schema PASS 100% 47 in → 94 out (80 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
tencent/hy3:medium PASS 100% 47 in → 150 out (140 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:medium__json_schema PASS 100% 47 in → 225 out (224 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
tencent/hy3:none PASS 100% 50 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:none__json_schema PASS 100% 50 in → 16 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
tencent/hy3:xhigh PASS 100% 47 in → 148 out (139 reasoning)
git submodule add ../lib-bare lib
tencent/hy3:xhigh__json_schema PASS 100% 47 in → 194 out (193 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling-small:high PASS 100% 51 in → 92 out (77 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:high__json_schema PASS 100% 51 in → 113 out (91 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling-small:low PASS 100% 51 in → 32 out (17 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:low__json_schema PASS 100% 51 in → 47 out (25 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling-small:max PASS 100% 51 in → 135 out (120 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:max__json_schema PASS 100% 51 in → 147 out (128 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
thinkingmachines/inkling-small:medium PASS 100% 51 in → 62 out (47 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:medium__json_schema PASS 100% 51 in → 104 out (85 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
thinkingmachines/inkling-small:none PASS 100% 49 in → 12 out (0 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:xhigh PASS 100% 51 in → 110 out (98 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 51 in → 148 out (127 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling:high PASS 100% 51 in → 95 out (82 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:low PASS 100% 51 in → 29 out (13 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:max PASS 100% 51 in → 49 out (34 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:max__json_schema PASS 100% 51 in → 89 out (68 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling:medium PASS 100% 51 in → 31 out (17 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:none PASS 100% 49 in → 13 out (0 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:none__json_schema PASS 100% 49 in → 24 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
thinkingmachines/inkling:xhigh PASS 100% 51 in → 90 out (77 reasoning)
git submodule add ../lib-bare lib
thinkingmachines/inkling:xhigh__json_schema PASS 100% 51 in → 60 out (41 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
x-ai/grok-4.3:high PASS 100% 220 in → 129 out (121 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:high__json_schema PASS 100% 276 in → 281 out (265 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
x-ai/grok-4.3:low PASS 100% 226 in → 157 out (149 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:low__json_schema PASS 100% 282 in → 264 out (252 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
x-ai/grok-4.3:max PASS 100% 220 in → 194 out (186 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:max__json_schema PASS 100% 276 in → 281 out (265 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
x-ai/grok-4.3:medium PASS 100% 226 in → 221 out (213 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:medium__json_schema PASS 100% 282 in → 344 out (328 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
x-ai/grok-4.3:none PASS 100% 218 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:none__json_schema PASS 100% 274 in → 12 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
x-ai/grok-4.3:xhigh PASS 100% 220 in → 176 out (168 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.3:xhigh__json_schema PASS 100% 276 in → 287 out (271 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
x-ai/grok-4.5:high PASS 100% 241 in → 177 out (169 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.5:high__json_schema PASS 100% 307 in → 354 out (341 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
x-ai/grok-4.5:low PASS 100% 241 in → 60 out (52 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.5:low__json_schema PASS 100% 307 in → 71 out (58 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
x-ai/grok-4.5:max PASS 100% 241 in → 124 out (116 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.5:max__json_schema PASS 100% 307 in → 221 out (208 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
x-ai/grok-4.5:medium PASS 100% 241 in → 114 out (106 reasoning)
git submodule add ../lib-bare lib
x-ai/grok-4.5:medium__json_schema PASS 100% 307 in → 186 out (173 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
x-ai/grok-4.5:xhigh PASS 100% 241 in → 129 out (117 reasoning)
``` git submodule add ../lib-bare lib ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 307 in → 275 out (262 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command":"git submodule add ../lib-bare lib"}
z-ai/glm-4.7-flash:high PASS 100% 40 in → 232 out (217 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-4.7-flash:low PASS 100% 40 in → 243 out (232 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-4.7-flash:medium PASS 100% 40 in → 285 out (265 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-4.7-flash:none PASS 100% 40 in → 8 out (0 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-4.7-flash:none__json_schema PASS 100% 40 in → 14 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
z-ai/glm-4.7-flash:xhigh PASS 100% 40 in → 290 out (280 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:high PASS 100% 47 in → 198 out (188 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:high__json_schema PASS 100% 47 in → 180 out (164 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
z-ai/glm-5.2:low PASS 100% 47 in → 245 out (241 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:low__json_schema PASS 100% 222 in → 55 out (37 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
z-ai/glm-5.2:medium PASS 100% 47 in → 164 out (153 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:medium__json_schema PASS 100% 47 in → 132 out (117 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
z-ai/glm-5.2:none PASS 100% 41 in → 9 out (0 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:none__json_schema PASS 100% 41 in → 14 out (0 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) {"command": "git submodule add ../lib-bare lib"}
z-ai/glm-5.2:xhigh PASS 100% 47 in → 250 out (240 reasoning)
git submodule add ../lib-bare lib
z-ai/glm-5.2:xhigh__json_schema PASS 100% 47 in → 339 out (321 reasoning)
git submodule add ../lib-bare lib
JSON Schema Structured Output
(raw) { "command": "git submodule add ../lib-bare lib" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low FAIL 0% 45 in → 309 out (337 reasoning)
gitsubmodule add ../lib-bare lib
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 45 in → 297 out (317 reasoning)
git submodule add../lib-bare lib
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
liquid/lfm-2-24b-a2b FAIL 0% 47 in → 16 out
git submodule add https://github.com/username/repo.git lib
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
Invalid JSON. Output: git submodule add https://github.com/username/repo.git lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high FAIL 0% 78 in → 401 out (438 reasoning)
submodule add ../lib-bare lib
Failure: Failed assertions: [{'type': 'file_exists', 'path': '.gitmodules'}, {'type': 'dir_exists', 'path': 'lib'}, {'type': 'git_output', 'command': 'git submodule status', 'contains': 'lib'}]
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8e5f36ab-83ab-9fc8-8cc5-f2db897f7d6d","request_id":"8e5f36ab-83ab-9fc8-8cc5-f2db897f7d6d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-46eb5fe6-ab57-9ca9-a3bb-66769397b151","request_id":"46eb5fe6-ab57-9ca9-a3bb-66769397b151"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3040103a-3170-9905-aadf-f4d7af09f586","request_id":"3040103a-3170-9905-aadf-f4d7af09f586"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-70733483-e384-9c30-9d61-b704076c23ad","request_id":"70733483-e384-9c30-9d61-b704076c23ad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7fcc0159-6019-9353-891d-248fc48004ae","request_id":"7fcc0159-6019-9353-891d-248fc48004ae"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-31f24d26-dce3-93bd-8430-2e85018270cb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c6fb7f0b-54b6-97a9-9b03-fbee19065593"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e5d3beb3-1b5b-9957-91ba-03219939c5f9"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-2fba938e-ed8c-93c9-822a-14147c1b6988"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f6c3c306-969e-98a9-bcad-8852a33adfd6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b2adbbdd-f705-9061-8f17-4ce5314db152"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-00b79d0d-bffe-9317-99df-c8831f0328fb","request_id":"00b79d0d-bffe-9317-99df-c8831f0328fb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a448ab01-d471-9d80-aa6d-96264e32d1f7","request_id":"a448ab01-d471-9d80-aa6d-96264e32d1f7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a364eadf-736f-9b39-bfcb-9818ac3ecd3b","request_id":"a364eadf-736f-9b39-bfcb-9818ac3ecd3b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-048578c7-856c-9c5c-9cbb-a858b79bad9d","request_id":"048578c7-856c-9c5c-9cbb-a858b79bad9d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-91e83490-0f9d-9c5c-83b8-e310271fdce1","request_id":"91e83490-0f9d-9c5c-83b8-e310271fdce1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bd34db2b-db6a-917f-9aa2-1e505eba1316","request_id":"bd34db2b-db6a-917f-9aa2-1e505eba1316"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-02efe9c2-23bb-96bd-ba03-dfd48fc0f76d","request_id":"02efe9c2-23bb-96bd-ba03-dfd48fc0f76d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14ef7456-899f-9966-b42f-ad689502a7f8","request_id":"14ef7456-899f-9966-b42f-ad689502a7f8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-49f300c1-ca61-96ac-addb-8437a9b834d9","request_id":"49f300c1-ca61-96ac-addb-8437a9b834d9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3d855161-16f3-950c-925a-4e9c2be86b14","request_id":"3d855161-16f3-950c-925a-4e9c2be86b14"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git submodule add ../lib-bare lib{ "command": "git submodule add ../lib-bare lib" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git submodule add ../lib-bare lib
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)