Create a signed tag (simulated with annotated tag)
Tests ability to create a signed tag (simulated with annotated tag). Evaluates signed-tag creation awareness.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 git config tag.gpgSign false
  5. 05 echo 'App' > app.txt
  6. 06 git add app.txt
  7. 07 git commit -m 'Secure release'
Prompt
Create an annotated git tag named 'v1.0-signed' on HEAD with the message 'Signed release'. Since no GPG key is configured, create it as a regular annotated tag. Output ONLY the git command, nothing else.
Expected
git tag -a v1.0-signed -m 'Signed release'

Scoped model quality, cost, API time, and token usage for tag_management/f009.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 105 in → 35 out (20 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
anthropic/claude-fable-5:high__json_schema PASS 100% 313 in → 34 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-fable-5:low PASS 100% 104 in → 33 out (24 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
anthropic/claude-fable-5:low__json_schema PASS 100% 316 in → 35 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
anthropic/claude-fable-5:max PASS 100% 103 in → 184 out (65 reasoning)
git tag -a v1.0-signed -m "Signed release"
anthropic/claude-fable-5:max__json_schema PASS 100% 312 in → 183 out (56 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
anthropic/claude-fable-5:medium PASS 100% 102 in → 35 out (20 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
anthropic/claude-fable-5:medium__json_schema PASS 100% 315 in → 35 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
anthropic/claude-fable-5:xhigh PASS 100% 101 in → 89 out (50 reasoning)
``` git tag -a v1.0-signed -m "Signed release" HEAD ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 314 in → 34 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-haiku-4.5:high PASS 100% 109 in → 410 out (358 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 272 in → 419 out (384 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-haiku-4.5:low PASS 100% 109 in → 191 out (152 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 271 in → 345 out (298 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
anthropic/claude-haiku-4.5:medium PASS 100% 111 in → 278 out (212 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 274 in → 256 out (219 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-haiku-4.5:none PASS 100% 81 in → 26 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' HEAD ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 243 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 111 in → 212 out (163 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 271 in → 234 out (192 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
anthropic/claude-opus-4.6:high PASS 100% 80 in → 72 out (27 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 245 in → 70 out (26 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
anthropic/claude-opus-4.6:low PASS 100% 79 in → 20 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
anthropic/claude-opus-4.6:low__json_schema PASS 100% 242 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.6:max PASS 100% 79 in → 113 out (67 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 242 in → 106 out (60 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.6:medium PASS 100% 83 in → 24 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 244 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.6:none PASS 100% 79 in → 24 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 243 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.6:xhigh PASS 100% 81 in → 65 out (25 reasoning)
git tag -a v1.0-signed -m "Signed release"
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 242 in → 68 out (25 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
anthropic/claude-opus-4.7:high PASS 100% 109 in → 24 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
anthropic/claude-opus-4.7:high__json_schema PASS 100% 320 in → 36 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.7:low PASS 100% 107 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
anthropic/claude-opus-4.7:low__json_schema PASS 100% 319 in → 32 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.7:max PASS 100% 108 in → 30 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 318 in → 36 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.7:medium PASS 100% 106 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 319 in → 36 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.7:none PASS 100% 108 in → 26 out (0 reasoning)
`git tag -a v1.0-signed -m "Signed release"`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 318 in → 36 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.7:xhigh PASS 100% 108 in → 29 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 318 in → 36 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-opus-4.8:high PASS 100% 104 in → 26 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 313 in → 30 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.8:low PASS 100% 104 in → 30 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" HEAD ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 313 in → 30 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.8:max PASS 100% 104 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
anthropic/claude-opus-4.8:max__json_schema PASS 100% 313 in → 30 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.8:medium PASS 100% 105 in → 27 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 315 in → 30 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.8:none PASS 100% 103 in → 31 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' HEAD ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 312 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-opus-4.8:xhigh PASS 100% 104 in → 27 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 315 in → 30 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-sonnet-4.6:high PASS 100% 81 in → 95 out (52 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 241 in → 115 out (68 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-4.6:low PASS 100% 78 in → 25 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 242 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-sonnet-4.6:max PASS 100% 80 in → 141 out (92 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 244 in → 153 out (107 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-sonnet-4.6:medium PASS 100% 83 in → 71 out (26 reasoning)
``` git tag -a v1.0-signed -m "Signed release" HEAD ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 242 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-4.6:none PASS 100% 81 in → 25 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 243 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 79 in → 114 out (69 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 242 in → 97 out (52 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-5:high PASS 100% 104 in → 30 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" HEAD ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 313 in → 34 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-5:low PASS 100% 103 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
anthropic/claude-sonnet-5:low__json_schema PASS 100% 314 in → 34 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-5:medium PASS 100% 104 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 314 in → 34 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-5:none PASS 100% 104 in → 30 out (0 reasoning)
``` git tag -a v1.0-signed -m "Signed release" HEAD ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 313 in → 33 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
anthropic/claude-sonnet-5:xhigh PASS 100% 101 in → 31 out (0 reasoning)
``` git tag -a v1.0-signed -m 'Signed release' HEAD ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 312 in → 35 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
arcee-ai/trinity-large-thinking:high PASS 100% 70 in → 332 out (322 reasoning)
git tag -a v1.0-signed -m 'Signed release'
arcee-ai/trinity-large-thinking:low PASS 100% 70 in → 304 out (288 reasoning)
git tag -a v1.0-signed -m "Signed release"
arcee-ai/trinity-large-thinking:medium PASS 100% 71 in → 696 out (689 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
arcee-ai/trinity-large-thinking:xhigh PASS 100% 69 in → 437 out (431 reasoning)
git tag -a v1.0-signed -m 'Signed release'
arcee-ai/trinity-mini:high__json_schema PASS 100% 71 in → 340 out (322 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
arcee-ai/trinity-mini:low PASS 100% 71 in → 354 out (341 reasoning)
git tag -a v1.0-signed -m "Signed release"
arcee-ai/trinity-mini:medium PASS 100% 69 in → 424 out (441 reasoning)
git tag -a v1.0-signed -m "Signed release"
arcee-ai/trinity-mini:medium__json_schema PASS 100% 71 in → 360 out (355 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
arcee-ai/trinity-mini:xhigh PASS 100% 70 in → 400 out (407 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-flash-0731:high PASS 100% 145 in → 75 out (58 reasoning)
git tag -a v1.0-signed -m 'Signed release'
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 144 in → 83 out (59 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 74 in → 82 out (75 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 65 in → 105 out (83 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
deepseek/deepseek-v4-flash-0731:max PASS 100% 158 in → 242 out (199 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 222 in → 147 out (121 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 68 in → 154 out (129 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 66 in → 143 out (112 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 65 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 66 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 67 in → 82 out (59 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 152 in → 94 out (71 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
deepseek/deepseek-v4-flash:high PASS 100% 65 in → 96 out (78 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 158 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
deepseek/deepseek-v4-flash:low PASS 100% 66 in → 91 out (73 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 66 in → 202 out (158 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
deepseek/deepseek-v4-flash:medium PASS 100% 67 in → 163 out (145 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 69 in → 229 out (203 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
deepseek/deepseek-v4-flash:none PASS 100% 67 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 67 in → 26 out (0 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed HEAD -m 'Signed release'" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 144 in → 287 out (269 reasoning)
git tag -a v1.0-signed -m 'Signed release'
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 145 in → 250 out (221 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
deepseek/deepseek-v4-pro:high PASS 100% 66 in → 146 out (128 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 253 in → 591 out (568 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
deepseek/deepseek-v4-pro:low PASS 100% 67 in → 194 out (171 reasoning)
```bash git tag -a v1.0-signed -m "Signed release" HEAD ```
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 66 in → 138 out (113 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\""}
deepseek/deepseek-v4-pro:medium PASS 100% 66 in → 136 out (118 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 254 in → 233 out (209 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
deepseek/deepseek-v4-pro:none PASS 100% 67 in → 22 out (0 reasoning)
```bash git tag -a v1.0-signed -m "Signed release" HEAD ```
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 66 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
deepseek/deepseek-v4-pro:xhigh PASS 100% 145 in → 194 out (176 reasoning)
git tag -a v1.0-signed -m "Signed release"
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 65 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
google/gemini-3-flash-preview:high PASS 100% 67 in → 264 out (248 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemini-3-flash-preview:high__json_schema PASS 100% 68 in → 305 out (284 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3-flash-preview:low PASS 100% 68 in → 193 out (177 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemini-3-flash-preview:low__json_schema PASS 100% 125 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
google/gemini-3-flash-preview:medium PASS 100% 68 in → 276 out (260 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemini-3-flash-preview:medium__json_schema PASS 100% 69 in → 277 out (256 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3-flash-preview:xhigh PASS 100% 68 in → 194 out (178 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 123 in → 200 out (180 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.1-flash-lite-preview:high PASS 100% 66 in → 342 out (326 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 69 in → 400 out (379 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 68 in → 141 out (125 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 124 in → 137 out (111 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 66 in → 151 out (135 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 124 in → 213 out (188 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 69 in → 238 out (222 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 125 in → 329 out (308 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.1-pro-preview:high PASS 100% 67 in → 248 out (232 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 125 in → 494 out (474 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.1-pro-preview:low PASS 100% 67 in → 234 out (218 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 124 in → 168 out (148 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.1-pro-preview:medium PASS 100% 67 in → 315 out (299 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 123 in → 108 out (88 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.1-pro-preview:xhigh PASS 100% 67 in → 248 out (232 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 124 in → 173 out (153 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.5-flash-lite:high PASS 100% 69 in → 352 out (336 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 124 in → 361 out (340 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
google/gemini-3.5-flash-lite:low PASS 100% 67 in → 17 out (0 reasoning)
`git tag -a v1.0-signed -m "Signed release"`
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 69 in → 27 out (0 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed HEAD -m \"Signed release\"" }
google/gemini-3.5-flash-lite:max PASS 100% 66 in → 337 out (321 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 69 in → 374 out (354 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
google/gemini-3.5-flash-lite:medium PASS 100% 68 in → 230 out (214 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 69 in → 366 out (345 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 69 in → 235 out (219 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 124 in → 242 out (221 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.5-flash:high PASS 100% 68 in → 373 out (357 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash:high__json_schema PASS 100% 123 in → 331 out (310 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.5-flash:low PASS 100% 67 in → 259 out (243 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash:low__json_schema PASS 100% 125 in → 126 out (105 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.5-flash:medium PASS 100% 67 in → 238 out (222 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash:medium__json_schema PASS 100% 124 in → 276 out (255 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.5-flash:xhigh PASS 100% 66 in → 315 out (299 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 125 in → 298 out (277 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.6-flash:high PASS 100% 68 in → 379 out (363 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.6-flash:high__json_schema PASS 100% 124 in → 345 out (319 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
google/gemini-3.6-flash:low PASS 100% 69 in → 180 out (164 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.6-flash:low__json_schema PASS 100% 125 in → 20 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.6-flash:max PASS 100% 68 in → 505 out (489 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.6-flash:max__json_schema PASS 100% 124 in → 312 out (292 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.6-flash:medium PASS 100% 68 in → 317 out (301 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.6-flash:medium__json_schema PASS 100% 125 in → 329 out (309 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemini-3.6-flash:xhigh PASS 100% 67 in → 458 out (442 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 124 in → 296 out (276 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
google/gemma-4-26b-a4b-it:high PASS 100% 83 in → 266 out (198 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 84 in → 267 out (189 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
google/gemma-4-26b-a4b-it:low PASS 100% 83 in → 293 out (276 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 84 in → 292 out (216 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
google/gemma-4-26b-a4b-it:medium PASS 100% 84 in → 275 out (210 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 84 in → 278 out (211 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
google/gemma-4-26b-a4b-it:none PASS 100% 80 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 79 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 82 in → 272 out (212 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 83 in → 288 out (211 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
google/gemma-4-31b-it:high PASS 100% 82 in → 167 out (122 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemma-4-31b-it:high__json_schema PASS 100% 84 in → 204 out (154 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemma-4-31b-it:low PASS 100% 82 in → 201 out (149 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemma-4-31b-it:low__json_schema PASS 100% 83 in → 168 out (113 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemma-4-31b-it:medium PASS 100% 84 in → 122 out (83 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemma-4-31b-it:medium__json_schema PASS 100% 83 in → 174 out (125 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
google/gemma-4-31b-it:none PASS 100% 79 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
google/gemma-4-31b-it:none__json_schema PASS 100% 80 in → 22 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
google/gemma-4-31b-it:xhigh PASS 100% 84 in → 163 out (118 reasoning)
git tag -a v1.0-signed -m 'Signed release'
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 84 in → 199 out (140 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
inclusionai/ling-2.6-flash PASS 100% 88 in → 23 out
```sh git tag -a v1.0-signed HEAD -m 'Signed release' ```
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed HEAD -m 'Signed release'" }
minimax/minimax-m2.5:high PASS 100% 100 in → 116 out (98 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m2.5:high__json_schema PASS 100% 100 in → 111 out (93 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
minimax/minimax-m2.5:low PASS 100% 99 in → 93 out (80 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m2.5:low__json_schema PASS 100% 100 in → 117 out (102 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
minimax/minimax-m2.5:medium PASS 100% 99 in → 124 out (108 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m2.5:medium__json_schema PASS 100% 100 in → 126 out (108 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
minimax/minimax-m2.5:xhigh PASS 100% 100 in → 124 out (119 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 99 in → 111 out (87 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
minimax/minimax-m2.7:high PASS 100% 99 in → 257 out (237 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
minimax/minimax-m2.7:high__json_schema PASS 100% 231 in → 367 out (344 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
minimax/minimax-m2.7:low PASS 100% 104 in → 342 out (337 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
minimax/minimax-m2.7:low__json_schema PASS 100% 98 in → 342 out (335 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
minimax/minimax-m2.7:medium PASS 100% 102 in → 334 out (317 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
minimax/minimax-m2.7:medium__json_schema PASS 100% 233 in → 281 out (259 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
minimax/minimax-m2.7:xhigh PASS 100% 101 in → 238 out (221 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
minimax/minimax-m3:high PASS 100% 237 in → 123 out (116 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m3:low PASS 100% 238 in → 127 out (115 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m3:low__json_schema PASS 100% 224 in → 24 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
minimax/minimax-m3:medium PASS 100% 239 in → 142 out (132 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m3:medium__json_schema PASS 100% 224 in → 22 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
minimax/minimax-m3:xhigh PASS 100% 233 in → 131 out (120 reasoning)
git tag -a v1.0-signed -m "Signed release"
minimax/minimax-m3:xhigh__json_schema PASS 100% 227 in → 22 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
mistralai/devstral-2512 PASS 100% 67 in → 23 out
```bash git tag -a v1.0-signed -m "Signed release" HEAD ```
mistralai/devstral-2512__json_schema PASS 100% 68 in → 23 out
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
mistralai/mistral-medium-3-5:high PASS 100% 79 in → 356 out (331 reasoning)
git tag -a v1.0-signed -m 'Signed release'
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 68 in → 282 out (252 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
mistralai/mistral-medium-3-5:low PASS 100% 79 in → 1,070 out (1,069 reasoning)
git tag -a v1.0-signed -m 'Signed release'
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 68 in → 416 out (388 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
mistralai/mistral-medium-3-5:medium PASS 100% 78 in → 360 out (346 reasoning)
git tag -a v1.0-signed -m "Signed release"
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 67 in → 461 out (437 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
mistralai/mistral-medium-3-5:none PASS 100% 80 in → 23 out (0 reasoning)
```bash git tag -a v1.0-signed -m "Signed release" HEAD ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 66 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 80 in → 320 out (313 reasoning)
git tag -a v1.0-signed -m "Signed release"
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 66 in → 444 out (436 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
moonshotai/kimi-k2.6:high PASS 100% 69 in → 557 out (575 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
moonshotai/kimi-k2.6:high__json_schema PASS 100% 251 in → 1,490 out (1,468 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
moonshotai/kimi-k2.6:low PASS 100% 69 in → 713 out (696 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
moonshotai/kimi-k2.6:low__json_schema PASS 100% 69 in → 776 out (794 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
moonshotai/kimi-k2.6:medium PASS 100% 69 in → 597 out (581 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 68 in → 798 out (249 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
moonshotai/kimi-k2.6:xhigh PASS 100% 69 in → 1,840 out (1,822 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 69 in → 641 out (422 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
moonshotai/kimi-k2.7-code:high PASS 100% 68 in → 101 out (84 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 69 in → 116 out (90 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
moonshotai/kimi-k2.7-code:low PASS 100% 69 in → 85 out (70 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 69 in → 151 out (127 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
moonshotai/kimi-k2.7-code:medium PASS 100% 71 in → 110 out (88 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 69 in → 92 out (70 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 68 in → 172 out (154 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 68 in → 158 out (134 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
moonshotai/kimi-k3:high PASS 100% 146 in → 155 out (125 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:high__json_schema PASS 100% 148 in → 108 out (78 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:low PASS 100% 147 in → 176 out (144 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
moonshotai/kimi-k3:low__json_schema PASS 100% 145 in → 221 out (190 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
moonshotai/kimi-k3:max PASS 100% 147 in → 119 out (89 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:max__json_schema PASS 100% 146 in → 106 out (76 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:medium PASS 100% 146 in → 78 out (48 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:medium__json_schema PASS 100% 148 in → 105 out (75 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:xhigh PASS 100% 147 in → 151 out (121 reasoning)
git tag -a v1.0-signed -m "Signed release"
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 146 in → 118 out (88 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 80 in → 193 out (166 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 81 in → 335 out (297 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 81 in → 263 out (258 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 81 in → 215 out (184 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 81 in → 92 out (71 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 80 in → 135 out (102 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\"" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 81 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 81 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 79 in → 3,009 out (3,266 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 80 in → 112 out (90 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 81 in → 66 out (38 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 81 in → 108 out (86 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 81 in → 143 out (114 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 81 in → 101 out (86 reasoning)
git tag -a v1.0-signed -m "Signed release"
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 81 in → 98 out (70 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 81 in → 18 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 80 in → 26 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 81 in → 154 out (124 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 80 in → 176 out (143 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed HEAD -m \"Signed release\"" }
openai/gpt-5.4-mini:high PASS 100% 69 in → 110 out (87 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.4-mini:high__json_schema PASS 100% 105 in → 197 out (166 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m 'Signed release'"}
openai/gpt-5.4-mini:low PASS 100% 69 in → 50 out (27 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.4-mini:low__json_schema PASS 100% 103 in → 85 out (54 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m 'Signed release'"}
openai/gpt-5.4-mini:medium PASS 100% 67 in → 104 out (81 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.4-mini:medium__json_schema PASS 100% 102 in → 282 out (251 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4-mini:none PASS 100% 68 in → 21 out (0 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.4-mini:none__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4-mini:xhigh PASS 100% 69 in → 194 out (171 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 104 in → 547 out (516 reasoning)
git tag -a -m "Signed release" v1.0-signed HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a -m \"Signed release\" v1.0-signed HEAD"}
openai/gpt-5.4-nano:high PASS 100% 68 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" 5eefa87
openai/gpt-5.4-nano:high__json_schema PASS 100% 103 in → 116 out (85 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m \"Signed release\""}
openai/gpt-5.4-nano:low PASS 100% 70 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.4-nano:low__json_schema PASS 100% 104 in → 28 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-5.4-nano:medium PASS 100% 68 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" 1e18cff
openai/gpt-5.4-nano:medium__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4-nano:none PASS 100% 69 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.4-nano:none__json_schema PASS 100% 102 in → 32 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" 3755764
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" 3755764"}
openai/gpt-5.4-nano:xhigh PASS 100% 69 in → 134 out (111 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 104 in → 654 out (623 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4:high PASS 100% 68 in → 100 out (78 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-5.4:high__json_schema PASS 100% 104 in → 226 out (195 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4:low PASS 100% 67 in → 72 out (49 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.4:low__json_schema PASS 100% 104 in → 63 out (32 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4:medium PASS 100% 69 in → 89 out (66 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.4:medium__json_schema PASS 100% 103 in → 105 out (74 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4:none PASS 100% 68 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.4:none__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.4:xhigh PASS 100% 69 in → 209 out (186 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.4:xhigh__json_schema PASS 100% 102 in → 387 out (356 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.5:high PASS 100% 67 in → 88 out (65 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.5:high__json_schema PASS 100% 104 in → 135 out (104 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.5:low PASS 100% 68 in → 56 out (33 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.5:low__json_schema PASS 100% 105 in → 29 out (0 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m \"Signed release\""}
openai/gpt-5.5:medium PASS 100% 67 in → 75 out (52 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.5:medium__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.5:none PASS 100% 68 in → 20 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-5.5:none__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.5:xhigh PASS 100% 69 in → 155 out (132 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.5:xhigh__json_schema PASS 100% 104 in → 158 out (127 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-luna:high PASS 100% 69 in → 76 out (53 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-luna:high__json_schema PASS 100% 104 in → 77 out (46 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m 'Signed release'"}
openai/gpt-5.6-luna:low PASS 100% 67 in → 57 out (34 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-luna:low__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-luna:max PASS 100% 67 in → 141 out (118 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.6-luna:max__json_schema PASS 100% 103 in → 547 out (516 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.6-luna:medium PASS 100% 67 in → 68 out (45 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
openai/gpt-5.6-luna:medium__json_schema PASS 100% 103 in → 60 out (29 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-luna:none PASS 100% 69 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-luna:none__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-luna:xhigh PASS 100% 69 in → 79 out (56 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 104 in → 79 out (48 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed HEAD -m \"Signed release\""}
openai/gpt-5.6-sol:high PASS 100% 67 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:high__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-sol:low PASS 100% 67 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:low__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-sol:max PASS 100% 68 in → 96 out (73 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:max__json_schema PASS 100% 106 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-sol:medium PASS 100% 69 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:medium__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.6-sol:none PASS 100% 68 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:none__json_schema PASS 100% 102 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-sol:xhigh PASS 100% 69 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-terra:high PASS 100% 69 in → 56 out (33 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:high__json_schema PASS 100% 102 in → 28 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-5.6-terra:low PASS 100% 71 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:low__json_schema PASS 100% 102 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-5.6-terra:max PASS 100% 69 in → 165 out (142 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:max__json_schema PASS 100% 103 in → 241 out (210 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.6-terra:medium PASS 100% 68 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:medium__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.6-terra:none PASS 100% 68 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:none__json_schema PASS 100% 103 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\" HEAD"}
openai/gpt-5.6-terra:xhigh PASS 100% 70 in → 57 out (34 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 104 in → 29 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
openai/gpt-oss-120b:high PASS 100% 133 in → 120 out (104 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-120b:high__json_schema PASS 100% 130 in → 285 out (252 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-oss-120b:low PASS 100% 131 in → 138 out (111 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-120b:low__json_schema PASS 100% 129 in → 75 out (44 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-oss-120b:medium PASS 100% 139 in → 136 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-120b:medium__json_schema PASS 100% 127 in → 128 out (86 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-oss-120b:xhigh PASS 100% 129 in → 196 out (165 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 116 in → 243 out (226 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-oss-20b:high PASS 100% 130 in → 91 out (74 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-oss-20b:high__json_schema PASS 100% 132 in → 337 out (292 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
openai/gpt-oss-20b:low PASS 100% 127 in → 51 out (20 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
openai/gpt-oss-20b:medium PASS 100% 117 in → 107 out (86 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-20b:medium__json_schema PASS 100% 129 in → 93 out (61 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
openai/gpt-oss-20b:xhigh PASS 100% 129 in → 695 out (656 reasoning)
git tag -a v1.0-signed -m "Signed release"
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 129 in → 20 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
poolside/laguna-m.1:high PASS 100% 80 in → 388 out (372 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-m.1:high__json_schema PASS 100% 77 in → 411 out (388 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
poolside/laguna-m.1:low PASS 100% 80 in → 309 out (293 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-m.1:low__json_schema PASS 100% 78 in → 271 out (247 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
poolside/laguna-m.1:medium PASS 100% 80 in → 338 out (320 reasoning)
git tag -a v1.0-signed -m 'Signed release'
poolside/laguna-m.1:medium__json_schema PASS 100% 80 in → 273 out (247 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
poolside/laguna-m.1:none PASS 100% 78 in → 17 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-m.1:none__json_schema PASS 100% 80 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
poolside/laguna-m.1:xhigh PASS 100% 78 in → 325 out (310 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-m.1:xhigh__json_schema PASS 100% 79 in → 334 out (314 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' HEAD"}
poolside/laguna-xs-2.1:high PASS 100% 78 in → 320 out (306 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-xs-2.1:high__json_schema PASS 100% 79 in → 201 out (179 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
poolside/laguna-xs-2.1:low PASS 100% 79 in → 295 out (281 reasoning)
git tag -a v1.0-signed -m 'Signed release'
poolside/laguna-xs-2.1:low__json_schema PASS 100% 80 in → 379 out (359 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\" HEAD"}
poolside/laguna-xs-2.1:medium PASS 100% 80 in → 303 out (289 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 77 in → 391 out (372 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
poolside/laguna-xs-2.1:none PASS 100% 79 in → 16 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
poolside/laguna-xs-2.1:none__json_schema PASS 100% 80 in → 25 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
poolside/laguna-xs-2.1:xhigh PASS 100% 80 in → 380 out (366 reasoning)
git tag -a v1.0-signed -m "Signed release"
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 80 in → 459 out (435 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
poolside/laguna-xs.2:high PASS 100% 117 in → 179 out (163 reasoning)
git tag -a v1.0-signed -m 'Signed release'
poolside/laguna-xs.2:high__json_schema PASS 100% 117 in → 188 out (168 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
poolside/laguna-xs.2:low PASS 100% 117 in → 279 out (265 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
poolside/laguna-xs.2:low__json_schema PASS 100% 115 in → 190 out (170 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
poolside/laguna-xs.2:medium PASS 100% 115 in → 152 out (136 reasoning)
git tag -a v1.0-signed -m 'Signed release'
poolside/laguna-xs.2:medium__json_schema PASS 100% 115 in → 259 out (239 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
poolside/laguna-xs.2:none PASS 100% 116 in → 23 out (0 reasoning)
```bash git tag -a v1.0-signed -m 'Signed release' HEAD ```
poolside/laguna-xs.2:none__json_schema PASS 100% 117 in → 22 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
poolside/laguna-xs.2:xhigh PASS 100% 115 in → 157 out (141 reasoning)
git tag -a v1.0-signed -m 'Signed release'
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 116 in → 243 out (222 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
qwen/qwen3.6-27b:high PASS 100% 76 in → 1,506 out (1,392 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
qwen/qwen3.6-27b:high__json_schema PASS 100% 76 in → 1,243 out (1,100 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
qwen/qwen3.6-27b:low PASS 100% 76 in → 1,364 out (1,213 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.6-27b:medium PASS 100% 77 in → 1,152 out (1,131 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
qwen/qwen3.6-27b:medium__json_schema PASS 100% 76 in → 524 out (1 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
qwen/qwen3.6-27b:none PASS 100% 78 in → 16 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.6-27b:none__json_schema PASS 100% 78 in → 27 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
qwen/qwen3.6-27b:xhigh PASS 100% 76 in → 1,034 out (1 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 77 in → 1,440 out (1,312 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
qwen/qwen3.6-35b-a3b:high PASS 100% 77 in → 792 out (714 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 76 in → 911 out (802 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
qwen/qwen3.6-35b-a3b:low PASS 100% 75 in → 914 out (808 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 77 in → 999 out (893 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 76 in → 379 out (318 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 77 in → 861 out (774 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
qwen/qwen3.6-35b-a3b:none PASS 100% 79 in → 16 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 76 in → 31 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" eec7be9
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" eec7be9" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 76 in → 1,165 out (1,072 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 77 in → 735 out (638 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\" HEAD" }
qwen/qwen3.6-flash:high PASS 100% 75 in → 862 out (841 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.6-flash:low PASS 100% 77 in → 1,045 out (1,024 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.6-flash:medium PASS 100% 77 in → 751 out (730 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
qwen/qwen3.6-flash:none PASS 100% 78 in → 15 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.6-flash:xhigh PASS 100% 76 in → 851 out (830 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-flash:high PASS 100% 76 in → 1,078 out (1,057 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-flash:low PASS 100% 77 in → 1,021 out (1,000 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-flash:max PASS 100% 75 in → 1,309 out (1,289 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.7-flash:medium PASS 100% 77 in → 1,035 out (1,014 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-flash:none PASS 100% 79 in → 15 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.7-flash:xhigh PASS 100% 76 in → 1,786 out (1,765 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
qwen/qwen3.7-max:high PASS 100% 77 in → 300 out (280 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.7-max:low PASS 100% 77 in → 1,120 out (1,099 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
qwen/qwen3.7-max:medium PASS 100% 77 in → 349 out (329 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.7-max:none PASS 100% 78 in → 15 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
qwen/qwen3.7-max:xhigh PASS 100% 77 in → 869 out (848 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-plus:high PASS 100% 77 in → 964 out (943 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-plus:low PASS 100% 76 in → 829 out (808 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-plus:medium PASS 100% 75 in → 1,014 out (993 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
qwen/qwen3.7-plus:none PASS 100% 78 in → 15 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
qwen/qwen3.7-plus:xhigh PASS 100% 77 in → 693 out (672 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
tencent/hy3:high PASS 100% 73 in → 666 out (648 reasoning)
git tag -a v1.0-signed -m "Signed release"
tencent/hy3:high__json_schema PASS 100% 75 in → 737 out (643 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
tencent/hy3:low PASS 100% 75 in → 292 out (263 reasoning)
git tag -a v1.0-signed -m "Signed release"
tencent/hy3:low__json_schema PASS 100% 74 in → 523 out (442 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
tencent/hy3:medium PASS 100% 76 in → 126 out (108 reasoning)
git tag -a v1.0-signed -m "Signed release"
tencent/hy3:medium__json_schema PASS 100% 75 in → 263 out (212 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
tencent/hy3:none PASS 100% 77 in → 18 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
tencent/hy3:none__json_schema PASS 100% 77 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
tencent/hy3:xhigh PASS 100% 74 in → 815 out (797 reasoning)
git tag -a v1.0-signed -m "Signed release"
tencent/hy3:xhigh__json_schema PASS 100% 73 in → 603 out (510 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
thinkingmachines/inkling-small:high PASS 100% 77 in → 175 out (145 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
thinkingmachines/inkling-small:high__json_schema PASS 100% 80 in → 148 out (120 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
thinkingmachines/inkling-small:low PASS 100% 78 in → 55 out (29 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
thinkingmachines/inkling-small:low__json_schema PASS 100% 77 in → 71 out (44 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release' HEAD"}
thinkingmachines/inkling-small:max PASS 100% 76 in → 161 out (139 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling-small:max__json_schema PASS 100% 77 in → 150 out (122 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
thinkingmachines/inkling-small:medium PASS 100% 78 in → 79 out (55 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
thinkingmachines/inkling-small:medium__json_schema PASS 100% 78 in → 183 out (153 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
thinkingmachines/inkling-small:none PASS 100% 77 in → 20 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
thinkingmachines/inkling-small:xhigh PASS 100% 77 in → 234 out (193 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 78 in → 231 out (203 reasoning)
git tag -a v1.0-signed HEAD -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed HEAD -m 'Signed release'" }
thinkingmachines/inkling:high PASS 100% 76 in → 251 out (228 reasoning)
git tag -a v1.0-signed HEAD -m "Signed release"
thinkingmachines/inkling:low PASS 100% 76 in → 43 out (16 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling:max PASS 100% 79 in → 84 out (62 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling:medium PASS 100% 76 in → 54 out (32 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling:medium__json_schema PASS 100% 76 in → 173 out (147 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
thinkingmachines/inkling:none PASS 100% 75 in → 19 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling:none__json_schema PASS 100% 75 in → 24 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'"}
thinkingmachines/inkling:xhigh PASS 100% 78 in → 232 out (207 reasoning)
git tag -a v1.0-signed -m "Signed release"
thinkingmachines/inkling:xhigh__json_schema PASS 100% 77 in → 165 out (140 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
x-ai/grok-4.3:high PASS 100% 246 in → 294 out (279 reasoning)
git tag -a v1.0-signed -m 'Signed release'
x-ai/grok-4.3:high__json_schema PASS 100% 301 in → 598 out (576 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
x-ai/grok-4.3:low PASS 100% 252 in → 265 out (250 reasoning)
git tag -a v1.0-signed -m "Signed release"
x-ai/grok-4.3:low__json_schema PASS 100% 308 in → 391 out (369 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
x-ai/grok-4.3:max PASS 100% 246 in → 441 out (426 reasoning)
git tag -a v1.0-signed -m "Signed release"
x-ai/grok-4.3:max__json_schema PASS 100% 302 in → 666 out (644 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m \"Signed release\"" }
x-ai/grok-4.3:medium PASS 100% 252 in → 394 out (379 reasoning)
git tag -a v1.0-signed -m "Signed release"
x-ai/grok-4.3:medium__json_schema PASS 100% 308 in → 623 out (601 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
x-ai/grok-4.3:none PASS 100% 243 in → 19 out (0 reasoning)
git tag -a v1.0-signed -m "Signed release" 1fdb420
x-ai/grok-4.3:none__json_schema PASS 100% 299 in → 23 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' 52af665
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release' 52af665"}
x-ai/grok-4.3:xhigh PASS 100% 247 in → 186 out (171 reasoning)
git tag -a v1.0-signed -m "Signed release"
x-ai/grok-4.3:xhigh__json_schema PASS 100% 302 in → 852 out (833 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
x-ai/grok-4.5:high PASS 100% 268 in → 235 out (216 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
x-ai/grok-4.5:high__json_schema PASS 100% 334 in → 167 out (145 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
x-ai/grok-4.5:low PASS 100% 268 in → 179 out (159 reasoning)
```bash git tag -a v1.0-signed -m "Signed release" ```
x-ai/grok-4.5:low__json_schema PASS 100% 333 in → 200 out (178 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
x-ai/grok-4.5:max PASS 100% 269 in → 218 out (202 reasoning)
git tag -a v1.0-signed -m "Signed release"
x-ai/grok-4.5:max__json_schema PASS 100% 334 in → 266 out (245 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m 'Signed release'"}
x-ai/grok-4.5:medium PASS 100% 268 in → 181 out (162 reasoning)
``` git tag -a v1.0-signed -m "Signed release" ```
x-ai/grok-4.5:medium__json_schema PASS 100% 333 in → 403 out (381 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
x-ai/grok-4.5:xhigh PASS 100% 269 in → 245 out (225 reasoning)
```bash git tag -a v1.0-signed -m 'Signed release' ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 334 in → 241 out (219 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
z-ai/glm-4.7-flash:high PASS 100% 66 in → 437 out (392 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
z-ai/glm-4.7-flash:high__json_schema PASS 100% 67 in → 466 out (413 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
z-ai/glm-4.7-flash:low PASS 100% 66 in → 459 out (394 reasoning)
git tag -a v1.0-signed -m "Signed release" HEAD
z-ai/glm-4.7-flash:low__json_schema PASS 100% 67 in → 450 out (397 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
z-ai/glm-4.7-flash:medium PASS 100% 67 in → 578 out (511 reasoning)
git tag -a v1.0-signed -m 'Signed release'
z-ai/glm-4.7-flash:none PASS 100% 67 in → 16 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
z-ai/glm-4.7-flash:none__json_schema PASS 100% 67 in → 21 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m 'Signed release'"}
z-ai/glm-4.7-flash:xhigh PASS 100% 67 in → 441 out (374 reasoning)
git tag -a v1.0-signed -m "Signed release"
z-ai/glm-5.2:high PASS 100% 65 in → 294 out (224 reasoning)
git tag -a v1.0-signed -m "Signed release"
z-ai/glm-5.2:low PASS 100% 73 in → 64 out (47 reasoning)
git tag -a v1.0-signed -m "Signed release"
z-ai/glm-5.2:low__json_schema PASS 100% 74 in → 77 out (55 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
z-ai/glm-5.2:medium PASS 100% 75 in → 311 out (294 reasoning)
git tag -a v1.0-signed -m 'Signed release'
z-ai/glm-5.2:medium__json_schema PASS 100% 250 in → 100 out (79 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0-signed -m \"Signed release\""}
z-ai/glm-5.2:none PASS 100% 67 in → 16 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
z-ai/glm-5.2:none__json_schema PASS 100% 67 in → 22 out (0 reasoning)
git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0-signed -m 'Signed release'" }
z-ai/glm-5.2:xhigh PASS 100% 75 in → 407 out (352 reasoning)
git tag -a v1.0-signed -m 'Signed release' HEAD
z-ai/glm-5.2:xhigh__json_schema PASS 100% 66 in → 316 out (241 reasoning)
git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0-signed -m \"Signed release\""}
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 69 in → 433 out (433 reasoning)
git tag -av1.0-signed -m "Signed release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
arcee-ai/trinity-mini:low__json_schema FAIL 0% 70 in → 324 out (310 reasoning)
git tag -av1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag -av1.0-signed -m \"Signed release\"" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 70 in → 366 out (357 reasoning)
git tag-a v1.0-signed -m "Signed release"
JSON Schema Structured Output
(raw) { "command": "git tag-a v1.0-signed -m \"Signed release\"" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
ibm-granite/granite-4.1-8b FAIL 0% 75 in → 14 out
git tag -as v1.0-signed "Signed release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
git tag -as v1.0-signed 'Signed release'
JSON Schema Structured Output
(raw) { "command": "git tag -as v1.0-signed 'Signed release'" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
liquid/lfm-2-24b-a2b FAIL 0% 74 in → 43 out
git tag -a v1.0-signed -m "Signed release" -s --sign-key /usr/share/keyrings/default-archive-keyring.gpg /dev/null
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
Invalid JSON. Output: git tag -a v1.0-signed -m "Signed release" --sign-key /usr/share/keyrings/default-archive-keyring.gpg --no-verify
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high__json_schema FAIL 0% 184 in → 61 out (56 reasoning)
:{
JSON Schema Structured Output
(raw) { "command":":{" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 81 in → 147 out (121 reasoning)
gittag -a v1.0-signed -m "Signed release" HEAD
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0-signed'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0-signed', 'contains': 'tag'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0-signed', 'contains': 'Signed release'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1ff32161-665d-9677-aa40-5fb59a2340b5","request_id":"1ff32161-665d-9677-aa40-5fb59a2340b5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3b94352d-2157-98a9-bb58-b55a458a72fc","request_id":"3b94352d-2157-98a9-bb58-b55a458a72fc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-89ab87b5-8ae6-9ce7-9a81-9acff379aa71","request_id":"89ab87b5-8ae6-9ce7-9a81-9acff379aa71"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4674f9e1-5458-9eeb-8c37-8e4cf6599a67","request_id":"4674f9e1-5458-9eeb-8c37-8e4cf6599a67"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6ee5cc28-80d7-9ae2-82d6-6fec9e3a7134","request_id":"6ee5cc28-80d7-9ae2-82d6-6fec9e3a7134"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-90366880-f05a-93ce-9ae5-90001e5f04e0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-662e7dbc-a2c8-9cbc-ae78-ac098f8b2dfb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6a932c04-76e0-9c30-a90d-49832782373b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1c970e98-73ea-9abc-9beb-e35f72e305f6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e7c1c2ec-6829-9e7b-949f-5d9a17a0707d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0d49db80-69fc-9b65-947b-c948a44f6766"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8115c99a-e406-9511-85a6-8ec9b0c1c528","request_id":"8115c99a-e406-9511-85a6-8ec9b0c1c528"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2991babf-bf72-94ef-ac8b-aa43a8183171","request_id":"2991babf-bf72-94ef-ac8b-aa43a8183171"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab52d82c-a432-97e3-94e2-313f5a9c027a","request_id":"ab52d82c-a432-97e3-94e2-313f5a9c027a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a15649f3-6b73-98f8-a3a5-aac4dfaac14e","request_id":"a15649f3-6b73-98f8-a3a5-aac4dfaac14e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b9f9acbe-8b73-9c1b-9a7e-8a2bfb538e7c","request_id":"b9f9acbe-8b73-9c1b-9a7e-8a2bfb538e7c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b59e6e29-108d-98ee-a8d8-f637711056cf","request_id":"b59e6e29-108d-98ee-a8d8-f637711056cf"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-20e931f6-3d7a-9234-a0a5-499b39ca2254","request_id":"20e931f6-3d7a-9234-a0a5-499b39ca2254"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e903b584-a399-9b87-a66b-dc0b45f3ba3a","request_id":"e903b584-a399-9b87-a66b-dc0b45f3ba3a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-66850db1-5fb0-919d-a23c-acf0255d1103","request_id":"66850db1-5fb0-919d-a23c-acf0255d1103"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4ae4dc23-dd72-942a-82ef-cbc4aba3a0b2","request_id":"4ae4dc23-dd72-942a-82ef-cbc4aba3a0b2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git tag -a v1.0-signed -m 'Signed release' HEAD{ "command": "git tag -a v1.0-signed -m 'Signed release' HEAD" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0-signed -m "Signed release" HEAD
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0-signed -m "Signed release"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0-signed -m 'Signed release'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)