Push a tag to a bare remote
Tests ability to push a tag to a bare remote repository. Evaluates tag-push workflow with remotes.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial release'
  7. 07 git tag v1.0
  8. 08 mkdir -p ../remote-bare && cd ../remote-bare && git init --bare
  9. 09 git remote add origin ../remote-bare
  10. 10 git push origin main
Prompt
Push the tag 'v1.0' to the remote 'origin'. Output ONLY the git command, nothing else.
Expected
git push origin v1.0

Scoped model quality, cost, API time, and token usage for tag_management/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 68 in → 20 out (4 reasoning)
``` git push origin v1.0 ```
anthropic/claude-fable-5:high__json_schema PASS 100% 277 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-fable-5:low PASS 100% 65 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-fable-5:low__json_schema PASS 100% 277 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-fable-5:max PASS 100% 67 in → 67 out (18 reasoning)
git push origin v1.0
anthropic/claude-fable-5:max__json_schema PASS 100% 278 in → 76 out (35 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-fable-5:medium PASS 100% 66 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 279 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-fable-5:xhigh PASS 100% 68 in → 24 out (13 reasoning)
git push origin v1.0
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 278 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-haiku-4.5:high PASS 100% 81 in → 136 out (110 reasoning)
``` git push origin v1.0 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 244 in → 185 out (154 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-haiku-4.5:low PASS 100% 81 in → 146 out (115 reasoning)
``` git push origin v1.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 246 in → 145 out (125 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 82 in → 150 out (122 reasoning)
``` git push origin v1.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 244 in → 156 out (136 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-haiku-4.5:none PASS 100% 52 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 214 in → 15 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 81 in → 139 out (114 reasoning)
``` git push origin v1.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 244 in → 155 out (135 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-opus-4.6:high PASS 100% 54 in → 15 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 217 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.6:low PASS 100% 53 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-opus-4.6:low__json_schema PASS 100% 215 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.6:max PASS 100% 52 in → 15 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 216 in → 51 out (16 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.6:medium PASS 100% 52 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 216 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.6:none PASS 100% 54 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 215 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.6:xhigh PASS 100% 52 in → 15 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 216 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:high PASS 100% 72 in → 14 out (0 reasoning)
`git push origin v1.0`
anthropic/claude-opus-4.7:high__json_schema PASS 100% 280 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:low PASS 100% 72 in → 14 out (0 reasoning)
`git push origin v1.0`
anthropic/claude-opus-4.7:low__json_schema PASS 100% 283 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:max PASS 100% 74 in → 17 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 283 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:medium PASS 100% 71 in → 14 out (0 reasoning)
`git push origin v1.0`
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 283 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:none PASS 100% 73 in → 17 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 282 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.7:xhigh PASS 100% 72 in → 17 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 283 in → 22 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.8:high PASS 100% 67 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 278 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.8:low PASS 100% 67 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-opus-4.8:low__json_schema PASS 100% 278 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.8:max PASS 100% 68 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-opus-4.8:max__json_schema PASS 100% 276 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.8:medium PASS 100% 67 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 277 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-opus-4.8:none PASS 100% 67 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 276 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-opus-4.8:xhigh PASS 100% 68 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 279 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:high PASS 100% 54 in → 50 out (16 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 216 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:low PASS 100% 55 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 216 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:max PASS 100% 54 in → 61 out (30 reasoning)
git push origin v1.0
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 215 in → 65 out (30 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:medium PASS 100% 53 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 217 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:none PASS 100% 51 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 216 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 53 in → 51 out (17 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 217 in → 14 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-5:high PASS 100% 68 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 277 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-5:low PASS 100% 66 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-sonnet-5:low__json_schema PASS 100% 278 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-5:medium PASS 100% 67 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 278 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
anthropic/claude-sonnet-5:none PASS 100% 66 in → 14 out (0 reasoning)
``` git push origin v1.0 ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 278 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
anthropic/claude-sonnet-5:xhigh PASS 100% 69 in → 10 out (0 reasoning)
git push origin v1.0
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 279 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
arcee-ai/trinity-large-thinking:high PASS 100% 49 in → 238 out (228 reasoning)
git push origin v1.0
arcee-ai/trinity-large-thinking:low PASS 100% 52 in → 196 out (187 reasoning)
git push origin v1.0
arcee-ai/trinity-large-thinking:medium PASS 100% 51 in → 313 out (304 reasoning)
git push origin v1.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 49 in → 265 out (255 reasoning)
git push origin v1.0
arcee-ai/trinity-mini:high PASS 100% 49 in → 243 out (252 reasoning)
git push origin v1.0
arcee-ai/trinity-mini:low PASS 100% 50 in → 252 out (256 reasoning)
git push origin v1.0
arcee-ai/trinity-mini:low__json_schema PASS 100% 51 in → 294 out (287 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
arcee-ai/trinity-mini:medium PASS 100% 50 in → 252 out (260 reasoning)
git push origin v1.0
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 51 in → 286 out (282 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 123 in → 40 out (28 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 122 in → 44 out (29 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 44 in → 20 out (12 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 43 in → 27 out (16 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 124 in → 41 out (33 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 138 in → 273 out (233 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 49 in → 24 out (18 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 47 in → 31 out (13 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 43 in → 8 out (0 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 50 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 44 in → 83 out (71 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:high PASS 100% 43 in → 45 out (36 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 45 in → 70 out (53 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-flash:low PASS 100% 45 in → 74 out (64 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:medium PASS 100% 46 in → 49 out (40 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 46 in → 112 out (96 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0"}
deepseek/deepseek-v4-flash:none PASS 100% 44 in → 8 out (0 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:xhigh PASS 100% 45 in → 57 out (48 reasoning)
git push origin v1.0
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 123 in → 187 out (168 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-pro:high PASS 100% 44 in → 85 out (76 reasoning)
git push origin v1.0
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 44 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-pro:low PASS 100% 45 in → 73 out (65 reasoning)
git push origin v1.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 45 in → 169 out (152 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
deepseek/deepseek-v4-pro:medium PASS 100% 45 in → 113 out (105 reasoning)
git push origin v1.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 231 in → 76 out (62 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
deepseek/deepseek-v4-pro:none PASS 100% 45 in → 7 out (0 reasoning)
git push origin v1.0
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 230 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 310 in → 343 out (329 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3-flash-preview:high PASS 100% 45 in → 121 out (114 reasoning)
git push origin v1.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 100 in → 76 out (64 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3-flash-preview:low PASS 100% 43 in → 113 out (106 reasoning)
git push origin v1.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 102 in → 123 out (112 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3-flash-preview:medium PASS 100% 45 in → 134 out (127 reasoning)
git push origin v1.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 102 in → 139 out (128 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 44 in → 116 out (109 reasoning)
git push origin v1.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 102 in → 210 out (199 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 45 in → 194 out (187 reasoning)
git push origin v1.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 45 in → 420 out (409 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 46 in → 146 out (139 reasoning)
git push origin v1.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 102 in → 117 out (105 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 44 in → 103 out (96 reasoning)
git push origin v1.0
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 45 in → 155 out (138 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 44 in → 248 out (241 reasoning)
git push origin v1.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 102 in → 204 out (192 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.1-pro-preview:high PASS 100% 44 in → 211 out (204 reasoning)
git push origin v1.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 102 in → 221 out (210 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.1-pro-preview:low PASS 100% 44 in → 213 out (206 reasoning)
git push origin v1.0
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 100 in → 197 out (185 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.1-pro-preview:medium PASS 100% 42 in → 219 out (212 reasoning)
git push origin v1.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 102 in → 89 out (77 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 44 in → 216 out (209 reasoning)
git push origin v1.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 101 in → 249 out (237 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.5-flash-lite:high PASS 100% 45 in → 232 out (225 reasoning)
git push origin v1.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 100 in → 231 out (220 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.5-flash-lite:low PASS 100% 46 in → 9 out (0 reasoning)
`git push origin v1.0`
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 101 in → 17 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemini-3.5-flash-lite:max PASS 100% 45 in → 257 out (250 reasoning)
git push origin v1.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 45 in → 261 out (250 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 46 in → 110 out (103 reasoning)
git push origin v1.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 102 in → 240 out (228 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 46 in → 224 out (217 reasoning)
git push origin v1.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 45 in → 242 out (230 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.5-flash:high PASS 100% 43 in → 189 out (182 reasoning)
git push origin v1.0
google/gemini-3.5-flash:high__json_schema PASS 100% 101 in → 263 out (252 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.5-flash:low PASS 100% 44 in → 220 out (213 reasoning)
git push origin v1.0
google/gemini-3.5-flash:low__json_schema PASS 100% 102 in → 65 out (53 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.5-flash:medium PASS 100% 44 in → 195 out (188 reasoning)
git push origin v1.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 101 in → 297 out (285 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.5-flash:xhigh PASS 100% 45 in → 225 out (218 reasoning)
git push origin v1.0
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 100 in → 313 out (301 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemini-3.6-flash:high PASS 100% 44 in → 151 out (144 reasoning)
git push origin v1.0
google/gemini-3.6-flash:high__json_schema PASS 100% 46 in → 255 out (238 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemini-3.6-flash:low PASS 100% 46 in → 129 out (122 reasoning)
git push origin v1.0
google/gemini-3.6-flash:low__json_schema PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.6-flash:max PASS 100% 46 in → 233 out (226 reasoning)
git push origin v1.0
google/gemini-3.6-flash:max__json_schema PASS 100% 102 in → 225 out (214 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.6-flash:medium PASS 100% 46 in → 188 out (181 reasoning)
git push origin v1.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 101 in → 219 out (208 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemini-3.6-flash:xhigh PASS 100% 45 in → 235 out (228 reasoning)
git push origin v1.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 102 in → 185 out (173 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 61 in → 134 out (126 reasoning)
git push origin v1.0
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 61 in → 139 out (98 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 59 in → 137 out (103 reasoning)
git push origin v1.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 60 in → 173 out (130 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 60 in → 165 out (128 reasoning)
git push origin v1.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 61 in → 173 out (128 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 55 in → 8 out (0 reasoning)
git push origin v1.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 58 in → 18 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 61 in → 121 out (92 reasoning)
git push origin v1.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 60 in → 133 out (99 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-31b-it:high PASS 100% 56 in → 120 out (87 reasoning)
git push origin v1.0
google/gemma-4-31b-it:high__json_schema PASS 100% 61 in → 159 out (123 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-31b-it:low PASS 100% 60 in → 134 out (96 reasoning)
git push origin v1.0
google/gemma-4-31b-it:low__json_schema PASS 100% 63 in → 199 out (186 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemma-4-31b-it:medium PASS 100% 61 in → 154 out (122 reasoning)
git push origin v1.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 62 in → 123 out (110 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
google/gemma-4-31b-it:none PASS 100% 57 in → 8 out (0 reasoning)
git push origin v1.0
google/gemma-4-31b-it:none__json_schema PASS 100% 58 in → 15 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
google/gemma-4-31b-it:xhigh PASS 100% 59 in → 129 out (98 reasoning)
git push origin v1.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 61 in → 540 out (278 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
ibm-granite/granite-4.1-8b PASS 100% 54 in → 8 out
git push origin v1.0
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
inclusionai/ling-2.6-flash PASS 100% 64 in → 12 out
``` git push origin v1.0 ```
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
liquid/lfm-2-24b-a2b PASS 100% 52 in → 14 out
```bash git push origin v1.0 ```
minimax/minimax-m2.5:high PASS 100% 85 in → 41 out (32 reasoning)
git push origin v1.0
minimax/minimax-m2.5:high__json_schema PASS 100% 79 in → 175 out (160 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m2.5:low PASS 100% 79 in → 66 out (60 reasoning)
git push origin v1.0
minimax/minimax-m2.5:low__json_schema PASS 100% 86 in → 164 out (144 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m2.5:medium PASS 100% 78 in → 69 out (61 reasoning)
git push origin v1.0
minimax/minimax-m2.5:medium__json_schema PASS 100% 78 in → 145 out (119 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m2.5:xhigh PASS 100% 77 in → 111 out (101 reasoning)
git push origin v1.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 78 in → 144 out (124 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m2.7:high PASS 100% 77 in → 71 out (58 reasoning)
git push origin v1.0
minimax/minimax-m2.7:low PASS 100% 78 in → 145 out (142 reasoning)
git push origin v1.0
minimax/minimax-m2.7:low__json_schema PASS 100% 79 in → 175 out (164 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
minimax/minimax-m2.7:medium PASS 100% 78 in → 223 out (247 reasoning)
git push origin v1.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 212 in → 233 out (220 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
minimax/minimax-m2.7:xhigh PASS 100% 81 in → 74 out (66 reasoning)
git push origin v1.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 212 in → 266 out (249 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m3:high PASS 100% 215 in → 43 out (31 reasoning)
git push origin v1.0
minimax/minimax-m3:high__json_schema PASS 100% 204 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
minimax/minimax-m3:low PASS 100% 216 in → 34 out (25 reasoning)
git push origin v1.0
minimax/minimax-m3:low__json_schema PASS 100% 163 in → 15 out (14 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
minimax/minimax-m3:medium PASS 100% 318 in → 35 out (24 reasoning)
git push origin v1.0
minimax/minimax-m3:medium__json_schema PASS 100% 203 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
minimax/minimax-m3:xhigh PASS 100% 217 in → 43 out (32 reasoning)
git push origin v1.0
minimax/minimax-m3:xhigh__json_schema PASS 100% 164 in → 16 out (15 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
mistralai/devstral-2512 PASS 100% 45 in → 13 out
```bash git push origin v1.0 ```
mistralai/devstral-2512__json_schema PASS 100% 45 in → 13 out
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
mistralai/mistral-medium-3-5:high PASS 100% 58 in → 198 out (177 reasoning)
git push origin v1.0
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 45 in → 250 out (246 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
mistralai/mistral-medium-3-5:low PASS 100% 57 in → 166 out (152 reasoning)
git push origin v1.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 45 in → 154 out (152 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 57 in → 177 out (152 reasoning)
git push origin v1.0
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 46 in → 178 out (183 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
mistralai/mistral-medium-3-5:none PASS 100% 56 in → 13 out (0 reasoning)
```bash git push origin v1.0 ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 46 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 57 in → 185 out (176 reasoning)
git push origin v1.0
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 44 in → 167 out (153 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
moonshotai/kimi-k2.6:high PASS 100% 47 in → 190 out (183 reasoning)
git push origin v1.0
moonshotai/kimi-k2.6:high__json_schema PASS 100% 46 in → 433 out (248 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
moonshotai/kimi-k2.6:low PASS 100% 49 in → 70 out (61 reasoning)
git push origin v1.0
moonshotai/kimi-k2.6:low__json_schema PASS 100% 49 in → 210 out (192 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
moonshotai/kimi-k2.6:medium PASS 100% 47 in → 124 out (115 reasoning)
git push origin v1.0
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 229 in → 531 out (518 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
moonshotai/kimi-k2.6:xhigh PASS 100% 50 in → 127 out (126 reasoning)
git push origin v1.0
moonshotai/kimi-k2.7-code:high PASS 100% 46 in → 69 out (60 reasoning)
git push origin v1.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 47 in → 66 out (49 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
moonshotai/kimi-k2.7-code:low PASS 100% 47 in → 51 out (42 reasoning)
git push origin v1.0
moonshotai/kimi-k2.7-code:medium PASS 100% 47 in → 82 out (73 reasoning)
git push origin v1.0
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 48 in → 119 out (99 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 47 in → 99 out (90 reasoning)
git push origin v1.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 132 in → 140 out (127 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
moonshotai/kimi-k3:high PASS 100% 125 in → 50 out (28 reasoning)
git push origin v1.0
moonshotai/kimi-k3:high__json_schema PASS 100% 125 in → 70 out (48 reasoning)
git push origin v1.0
moonshotai/kimi-k3:low PASS 100% 125 in → 86 out (64 reasoning)
git push origin v1.0
moonshotai/kimi-k3:low__json_schema PASS 100% 221 in → 64 out (37 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
moonshotai/kimi-k3:max PASS 100% 125 in → 50 out (28 reasoning)
git push origin v1.0
moonshotai/kimi-k3:max__json_schema PASS 100% 124 in → 60 out (38 reasoning)
git push origin v1.0
moonshotai/kimi-k3:medium PASS 100% 125 in → 64 out (42 reasoning)
git push origin v1.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 124 in → 74 out (52 reasoning)
git push origin v1.0
moonshotai/kimi-k3:xhigh PASS 100% 126 in → 75 out (53 reasoning)
git push origin v1.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 125 in → 81 out (59 reasoning)
git push origin v1.0
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 58 in → 70 out (60 reasoning)
git push origin v1.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 58 in → 54 out (37 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 59 in → 65 out (52 reasoning)
git push origin v1.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 58 in → 58 out (37 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 59 in → 139 out (130 reasoning)
git push origin v1.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 58 in → 107 out (90 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 57 in → 17 out (0 reasoning)
git tag -f v1.0 && git push origin v1.0
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 57 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 59 in → 92 out (70 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 58 in → 66 out (50 reasoning)
git push origin v1.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 58 in → 106 out (89 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 59 in → 44 out (34 reasoning)
git push origin v1.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 58 in → 52 out (35 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 58 in → 81 out (70 reasoning)
git push origin v1.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 58 in → 80 out (62 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 59 in → 8 out (0 reasoning)
git push origin v1.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 57 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 59 in → 97 out (79 reasoning)
git push origin v1.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 58 in → 72 out (55 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
openai/gpt-5.4-mini:high PASS 100% 45 in → 195 out (182 reasoning)
git push origin v1.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 81 in → 149 out (128 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-mini:low PASS 100% 46 in → 31 out (18 reasoning)
git push origin v1.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 81 in → 67 out (46 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-mini:medium PASS 100% 46 in → 54 out (41 reasoning)
git push origin v1.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 82 in → 120 out (99 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-mini:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-mini:xhigh PASS 100% 45 in → 115 out (102 reasoning)
git push origin v1.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 80 in → 537 out (516 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-nano:high PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-nano:high__json_schema PASS 100% 82 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-nano:low PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-nano:low__json_schema PASS 100% 83 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-nano:medium PASS 100% 48 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-nano:medium__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-nano:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4-nano:xhigh PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 82 in → 610 out (589 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4:high PASS 100% 47 in → 56 out (43 reasoning)
git push origin v1.0
openai/gpt-5.4:high__json_schema PASS 100% 81 in → 165 out (144 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4:low PASS 100% 46 in → 22 out (9 reasoning)
git push origin v1.0
openai/gpt-5.4:low__json_schema PASS 100% 82 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4:medium PASS 100% 46 in → 59 out (46 reasoning)
git push origin v1.0
openai/gpt-5.4:medium__json_schema PASS 100% 83 in → 31 out (10 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.4:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.4:xhigh PASS 100% 45 in → 63 out (50 reasoning)
git push origin v1.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 82 in → 537 out (516 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.5:high PASS 100% 46 in → 46 out (33 reasoning)
git push origin v1.0
openai/gpt-5.5:high__json_schema PASS 100% 81 in → 103 out (82 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.5:low PASS 100% 48 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.5:low__json_schema PASS 100% 80 in → 20 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
openai/gpt-5.5:medium PASS 100% 46 in → 33 out (20 reasoning)
git push origin v1.0
openai/gpt-5.5:medium__json_schema PASS 100% 82 in → 20 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
openai/gpt-5.5:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.5:none__json_schema PASS 100% 82 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.5:xhigh PASS 100% 48 in → 48 out (35 reasoning)
git push origin v1.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 81 in → 158 out (137 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:high PASS 100% 48 in → 31 out (18 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 80 in → 105 out (84 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:low PASS 100% 47 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:max PASS 100% 46 in → 48 out (35 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 81 in → 150 out (129 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:medium PASS 100% 48 in → 31 out (18 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-luna:xhigh PASS 100% 45 in → 38 out (25 reasoning)
git push origin v1.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 83 in → 330 out (309 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:high PASS 100% 47 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:low PASS 100% 47 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:max PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:medium PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:none PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 83 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-sol:xhigh PASS 100% 45 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:high PASS 100% 49 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:low PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 80 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:max PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:medium PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 83 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:none PASS 100% 48 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 82 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 46 in → 11 out (0 reasoning)
git push origin v1.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 81 in → 19 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-oss-120b:high PASS 100% 97 in → 422 out (357 reasoning)
git push origin v1.0
openai/gpt-oss-120b:high__json_schema PASS 100% 95 in → 252 out (214 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-oss-120b:low PASS 100% 94 in → 28 out (9 reasoning)
git push origin v1.0
openai/gpt-oss-120b:low__json_schema PASS 100% 94 in → 42 out (17 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-oss-120b:medium__json_schema PASS 100% 105 in → 119 out (88 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
openai/gpt-oss-120b:xhigh PASS 100% 107 in → 44 out (37 reasoning)
git push origin v1.0
openai/gpt-oss-20b:high PASS 100% 112 in → 83 out (60 reasoning)
git push origin v1.0
openai/gpt-oss-20b:high__json_schema PASS 100% 109 in → 77 out (53 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
openai/gpt-oss-20b:low PASS 100% 106 in → 28 out (8 reasoning)
git push origin v1.0
openai/gpt-oss-20b:low__json_schema PASS 100% 106 in → 11 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-oss-20b:medium PASS 100% 108 in → 61 out (43 reasoning)
git push origin v1.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 109 in → 11 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
openai/gpt-oss-20b:xhigh PASS 100% 107 in → 125 out (102 reasoning)
git push origin v1.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 93 in → 295 out (265 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-m.1:high PASS 100% 58 in → 271 out (261 reasoning)
git push origin v1.0
poolside/laguna-m.1:high__json_schema PASS 100% 59 in → 285 out (272 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-m.1:low PASS 100% 57 in → 237 out (229 reasoning)
git push origin v1.0
poolside/laguna-m.1:low__json_schema PASS 100% 59 in → 278 out (260 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-m.1:medium PASS 100% 58 in → 296 out (287 reasoning)
git push origin v1.0
poolside/laguna-m.1:medium__json_schema PASS 100% 58 in → 255 out (239 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-m.1:none PASS 100% 58 in → 9 out (0 reasoning)
git push origin v1.0
poolside/laguna-m.1:none__json_schema PASS 100% 59 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-m.1:xhigh PASS 100% 57 in → 226 out (216 reasoning)
git push origin v1.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 59 in → 341 out (324 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-xs-2.1:high PASS 100% 59 in → 254 out (248 reasoning)
git push origin v1.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 59 in → 294 out (282 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs-2.1:low PASS 100% 59 in → 293 out (286 reasoning)
git push origin v1.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 59 in → 310 out (298 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs-2.1:medium PASS 100% 59 in → 228 out (220 reasoning)
git push origin v1.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 59 in → 240 out (227 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs-2.1:none PASS 100% 57 in → 7 out (0 reasoning)
git push origin v1.0
poolside/laguna-xs-2.1:none__json_schema PASS 100% 58 in → 18 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-xs-2.1:xhigh PASS 100% 59 in → 257 out (251 reasoning)
git push origin v1.0
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 58 in → 228 out (215 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs.2:high PASS 100% 96 in → 65 out (55 reasoning)
git push origin v1.0
poolside/laguna-xs.2:high__json_schema PASS 100% 96 in → 82 out (68 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs.2:low PASS 100% 95 in → 109 out (100 reasoning)
git push origin v1.0
poolside/laguna-xs.2:low__json_schema PASS 100% 96 in → 79 out (66 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs.2:medium PASS 100% 94 in → 45 out (35 reasoning)
git push origin v1.0
poolside/laguna-xs.2:medium__json_schema PASS 100% 96 in → 118 out (100 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
poolside/laguna-xs.2:none PASS 100% 96 in → 9 out (0 reasoning)
git push origin v1.0
poolside/laguna-xs.2:none__json_schema PASS 100% 96 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
poolside/laguna-xs.2:xhigh PASS 100% 96 in → 71 out (61 reasoning)
git push origin v1.0
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 95 in → 114 out (96 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-27b:high PASS 100% 54 in → 339 out (319 reasoning)
git push origin v1.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 53 in → 322 out (1 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-27b:low PASS 100% 55 in → 296 out (258 reasoning)
git push origin v1.0
qwen/qwen3.6-27b:medium PASS 100% 53 in → 561 out (531 reasoning)
git push origin v1.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 52 in → 1,064 out (982 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-27b:none PASS 100% 56 in → 8 out (0 reasoning)
git push origin v1.0
qwen/qwen3.6-27b:none__json_schema PASS 100% 56 in → 18 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-27b:xhigh PASS 100% 52 in → 319 out (294 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:high PASS 100% 54 in → 251 out (227 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 53 in → 646 out (590 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 55 in → 239 out (215 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 55 in → 692 out (643 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 55 in → 567 out (520 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 54 in → 247 out (214 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 57 in → 7 out (0 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 56 in → 18 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 54 in → 617 out (555 reasoning)
git push origin v1.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 54 in → 280 out (240 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
qwen/qwen3.6-flash:high PASS 100% 55 in → 814 out (802 reasoning)
git push origin v1.0
qwen/qwen3.6-flash:low PASS 100% 55 in → 696 out (684 reasoning)
git push origin v1.0
qwen/qwen3.6-flash:medium PASS 100% 55 in → 946 out (934 reasoning)
git push origin v1.0
qwen/qwen3.6-flash:none PASS 100% 55 in → 7 out (0 reasoning)
git push origin v1.0
qwen/qwen3.6-flash:xhigh PASS 100% 54 in → 761 out (749 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:high PASS 100% 55 in → 240 out (228 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:low PASS 100% 52 in → 256 out (244 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:low__json_schema PASS 100% 55 in → 684 out (672 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:max PASS 100% 54 in → 287 out (275 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:medium PASS 100% 53 in → 621 out (609 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:none PASS 100% 56 in → 7 out (0 reasoning)
git push origin v1.0
qwen/qwen3.7-flash:xhigh PASS 100% 54 in → 202 out (190 reasoning)
git push origin v1.0
qwen/qwen3.7-max:high PASS 100% 53 in → 738 out (726 reasoning)
git push origin v1.0
qwen/qwen3.7-max:low PASS 100% 54 in → 460 out (448 reasoning)
git push origin v1.0
qwen/qwen3.7-max:medium PASS 100% 55 in → 595 out (583 reasoning)
git push origin v1.0
qwen/qwen3.7-max:none PASS 100% 56 in → 7 out (0 reasoning)
git push origin v1.0
qwen/qwen3.7-max:xhigh PASS 100% 54 in → 483 out (471 reasoning)
git push origin v1.0
qwen/qwen3.7-plus:high PASS 100% 55 in → 254 out (242 reasoning)
git push origin v1.0
qwen/qwen3.7-plus:low PASS 100% 55 in → 436 out (424 reasoning)
git push origin v1.0
qwen/qwen3.7-plus:medium PASS 100% 55 in → 379 out (367 reasoning)
git push origin v1.0
qwen/qwen3.7-plus:none PASS 100% 56 in → 7 out (0 reasoning)
git push origin v1.0
qwen/qwen3.7-plus:xhigh PASS 100% 55 in → 633 out (621 reasoning)
git push origin v1.0
tencent/hy3:high PASS 100% 52 in → 168 out (159 reasoning)
git push origin v1.0
tencent/hy3:high__json_schema PASS 100% 53 in → 103 out (77 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
tencent/hy3:low PASS 100% 53 in → 192 out (183 reasoning)
git push origin v1.0
tencent/hy3:low__json_schema PASS 100% 51 in → 89 out (64 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
tencent/hy3:medium PASS 100% 51 in → 150 out (141 reasoning)
git push origin v1.0
tencent/hy3:medium__json_schema PASS 100% 52 in → 181 out (151 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
tencent/hy3:none PASS 100% 54 in → 8 out (0 reasoning)
git push origin v1.0
tencent/hy3:none__json_schema PASS 100% 56 in → 15 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
tencent/hy3:xhigh PASS 100% 51 in → 119 out (110 reasoning)
git push origin v1.0
tencent/hy3:xhigh__json_schema PASS 100% 53 in → 174 out (143 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling-small:high PASS 100% 55 in → 68 out (46 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 56 in → 75 out (58 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
thinkingmachines/inkling-small:low PASS 100% 56 in → 27 out (14 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 56 in → 34 out (14 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling-small:max PASS 100% 56 in → 120 out (90 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 56 in → 63 out (44 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling-small:medium PASS 100% 54 in → 95 out (75 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 56 in → 120 out (104 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
thinkingmachines/inkling-small:none PASS 100% 53 in → 10 out (0 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:xhigh PASS 100% 54 in → 70 out (57 reasoning)
git push origin v1.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 54 in → 62 out (42 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling:high PASS 100% 55 in → 80 out (60 reasoning)
git push origin v1.0
thinkingmachines/inkling:high__json_schema PASS 100% 57 in → 51 out (31 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling:low PASS 100% 55 in → 25 out (11 reasoning)
git push origin v1.0
thinkingmachines/inkling:max PASS 100% 56 in → 28 out (15 reasoning)
git push origin v1.0
thinkingmachines/inkling:medium PASS 100% 57 in → 28 out (15 reasoning)
git push origin v1.0
thinkingmachines/inkling:medium__json_schema PASS 100% 55 in → 85 out (66 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
thinkingmachines/inkling:none PASS 100% 54 in → 10 out (0 reasoning)
git push origin v1.0
thinkingmachines/inkling:xhigh PASS 100% 55 in → 39 out (26 reasoning)
git push origin v1.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 55 in → 82 out (63 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
x-ai/grok-4.3:high PASS 100% 225 in → 214 out (207 reasoning)
git push origin v1.0
x-ai/grok-4.3:high__json_schema PASS 100% 281 in → 258 out (247 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.3:low PASS 100% 232 in → 103 out (96 reasoning)
git push origin v1.0
x-ai/grok-4.3:low__json_schema PASS 100% 287 in → 232 out (217 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
x-ai/grok-4.3:max PASS 100% 224 in → 99 out (92 reasoning)
git push origin v1.0
x-ai/grok-4.3:max__json_schema PASS 100% 281 in → 285 out (274 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.3:medium PASS 100% 232 in → 192 out (185 reasoning)
git push origin v1.0
x-ai/grok-4.3:medium__json_schema PASS 100% 288 in → 246 out (231 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
x-ai/grok-4.3:none PASS 100% 223 in → 7 out (0 reasoning)
git push origin v1.0
x-ai/grok-4.3:none__json_schema PASS 100% 280 in → 11 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.3:xhigh PASS 100% 224 in → 90 out (83 reasoning)
git push origin v1.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 281 in → 280 out (265 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
x-ai/grok-4.5:high PASS 100% 245 in → 121 out (114 reasoning)
git push origin v1.0
x-ai/grok-4.5:high__json_schema PASS 100% 311 in → 292 out (280 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.5:low PASS 100% 247 in → 96 out (89 reasoning)
git push origin v1.0
x-ai/grok-4.5:low__json_schema PASS 100% 312 in → 230 out (218 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.5:max PASS 100% 246 in → 122 out (115 reasoning)
git push origin v1.0
x-ai/grok-4.5:max__json_schema PASS 100% 313 in → 192 out (180 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.5:medium PASS 100% 246 in → 129 out (122 reasoning)
git push origin v1.0
x-ai/grok-4.5:medium__json_schema PASS 100% 312 in → 200 out (188 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
x-ai/grok-4.5:xhigh PASS 100% 246 in → 90 out (79 reasoning)
``` git push origin v1.0 ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 311 in → 290 out (278 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command":"git push origin v1.0"}
z-ai/glm-4.7-flash:high PASS 100% 45 in → 246 out (218 reasoning)
git push origin v1.0
z-ai/glm-4.7-flash:low PASS 100% 47 in → 227 out (182 reasoning)
git push origin v1.0
z-ai/glm-4.7-flash:low__json_schema PASS 100% 46 in → 263 out (226 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
z-ai/glm-4.7-flash:medium PASS 100% 45 in → 286 out (251 reasoning)
git push origin v1.0
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 47 in → 290 out (256 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
z-ai/glm-4.7-flash:none PASS 100% 46 in → 7 out (0 reasoning)
git push origin v1.0
z-ai/glm-4.7-flash:none__json_schema PASS 100% 45 in → 13 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
z-ai/glm-4.7-flash:xhigh PASS 100% 47 in → 263 out (231 reasoning)
git push origin v1.0
z-ai/glm-5.2:high PASS 100% 46 in → 213 out (171 reasoning)
git push origin v1.0
z-ai/glm-5.2:low PASS 100% 52 in → 167 out (144 reasoning)
git push origin v1.0
z-ai/glm-5.2:low__json_schema PASS 100% 48 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
z-ai/glm-5.2:medium PASS 100% 52 in → 102 out (93 reasoning)
git push origin v1.0
z-ai/glm-5.2:medium__json_schema PASS 100% 228 in → 80 out (63 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
z-ai/glm-5.2:none PASS 100% 47 in → 8 out (0 reasoning)
git push origin v1.0
z-ai/glm-5.2:none__json_schema PASS 100% 47 in → 16 out (0 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git push origin v1.0" }
z-ai/glm-5.2:xhigh PASS 100% 52 in → 217 out (208 reasoning)
git push origin v1.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 226 in → 241 out (227 reasoning)
git push origin v1.0
JSON Schema Structured Output
(raw) {"command": "git push origin v1.0"}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high__json_schema FAIL 0% 49 in → 298 out (296 reasoning)
git push originv1.0
JSON Schema Structured Output
(raw) { "command": "git push originv1.0" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 49 in → 249 out (248 reasoning)
git pushorigin v1.0
JSON Schema Structured Output
(raw) { "command": "git pushorigin v1.0" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 50 in → 278 out (283 reasoning)
gitpush origin v1.0
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 130 in → 94 out (70 reasoning)
git push -v1.0
JSON Schema Structured Output
(raw) {"command":"git push -v1.0" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
Invalid JSON. Output: git push origin v1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 46 in → 29 out (0 reasoning)
git tag v1.0 -m "Initial release" && git push origin v1.0
JSON Schema Structured Output
(raw) { "command": "git tag v1.0 -m \"Initial release\" && git push origin v1.0" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
deepseek/deepseek-v4-pro:xhigh FAIL 0% 43 in → 48 out (34 reasoning)
git push originv1.0
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
Invalid JSON. Output: ```bash git push origin v1.0 ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git push origin v1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 59 in → 78 out (72 reasoning)
gitpush origin v1.0
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
openai/gpt-oss-120b:medium FAIL 0% 110 in → 58 out (37 reasoning)
git push origin v1.
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 106 in → 156 out (148 reasoning)
final
JSON Schema Structured Output
(raw) {"command":"final"}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git ls-remote --tags origin', 'contains': 'v1.0'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-db48fa12-a2ee-9583-994e-836247b10bbd","request_id":"db48fa12-a2ee-9583-994e-836247b10bbd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-340bcf2d-a678-9969-8fe6-f85de1b61b43","request_id":"340bcf2d-a678-9969-8fe6-f85de1b61b43"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-87b9fb2e-bd93-95e3-950e-4f8a54d8c411","request_id":"87b9fb2e-bd93-95e3-950e-4f8a54d8c411"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-451bcd9a-b9f9-95b4-bc2b-95cc5323b253","request_id":"451bcd9a-b9f9-95b4-bc2b-95cc5323b253"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-39bbbd23-69c8-9b0b-9fd6-0e7232fc3220","request_id":"39bbbd23-69c8-9b0b-9fd6-0e7232fc3220"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-403ba099-3066-9c86-8c51-a354d54c4913"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-fa8ec7da-395d-98f1-af5a-15296a13f36d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-be231a33-8b15-9d73-8756-2b35ce8e0acb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a8552d5c-57db-936d-a646-372ea748b84c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4ffbbe41-3291-9772-820e-ddb00a3b8d6f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d19a1403-be21-927f-a2a9-b72915ec3242","request_id":"d19a1403-be21-927f-a2a9-b72915ec3242"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab1e8f81-fc6c-9289-952f-bb7953184ff9","request_id":"ab1e8f81-fc6c-9289-952f-bb7953184ff9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bc6a1406-7ca4-98de-8e93-b64df89dd86e","request_id":"bc6a1406-7ca4-98de-8e93-b64df89dd86e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0b429d7c-9aca-91e3-ac34-445f21adb3f1","request_id":"0b429d7c-9aca-91e3-ac34-445f21adb3f1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5d70032f-0f35-9208-b433-92df8cda2c4d","request_id":"5d70032f-0f35-9208-b433-92df8cda2c4d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-109d99f9-282e-934c-87e9-f81066f298d3","request_id":"109d99f9-282e-934c-87e9-f81066f298d3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3fcff1c5-6911-90bc-bf2d-23c66a574893","request_id":"3fcff1c5-6911-90bc-bf2d-23c66a574893"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-248710fd-e1b8-9f1d-a495-0a0a6fe323ef","request_id":"248710fd-e1b8-9f1d-a495-0a0a6fe323ef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8479deb5-778b-9f7c-99a2-efd2c8136803","request_id":"8479deb5-778b-9f7c-99a2-efd2c8136803"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2a97830c-05c9-9cb3-b87d-6bb50557eab5","request_id":"2a97830c-05c9-9cb3-b87d-6bb50557eab5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git push origin v1.0{"command": "git push origin v1.0"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git push origin v1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git push origin v1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git push origin v1.0{ "command": "git push origin v1.0" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git push origin v1.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)