Create an annotated tag with a message
Tests ability to create an annotated tag with a message. Evaluates annotated-tag creation with metadata.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial release'
Prompt
Create an annotated git tag named 'v1.0.0' on HEAD with the message 'First stable release'. Output ONLY the git command, nothing else.
Expected
git tag -a v1.0.0 -m 'First stable release'

Scoped model quality, cost, API time, and token usage for tag_management/f002.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 78 in → 30 out (20 reasoning)
git tag -a v1.0.0 -m "First stable release"
anthropic/claude-fable-5:high__json_schema PASS 100% 288 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-fable-5:low PASS 100% 80 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-fable-5:low__json_schema PASS 100% 291 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-fable-5:max PASS 100% 80 in → 138 out (45 reasoning)
git tag -a v1.0.0 -m "First stable release"
anthropic/claude-fable-5:max__json_schema PASS 100% 290 in → 173 out (38 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
anthropic/claude-fable-5:medium PASS 100% 78 in → 30 out (14 reasoning)
git tag -a v1.0.0 -m "First stable release"
anthropic/claude-fable-5:medium__json_schema PASS 100% 290 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-fable-5:xhigh PASS 100% 78 in → 33 out (13 reasoning)
git tag -a v1.0.0 -m "First stable release"
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 291 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-haiku-4.5:high PASS 100% 90 in → 194 out (143 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 254 in → 499 out (431 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-haiku-4.5:low PASS 100% 92 in → 148 out (99 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 255 in → 185 out (137 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
anthropic/claude-haiku-4.5:medium PASS 100% 92 in → 172 out (122 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 253 in → 277 out (227 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-haiku-4.5:none PASS 100% 61 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 224 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 91 in → 178 out (138 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 253 in → 433 out (385 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
anthropic/claude-opus-4.6:high PASS 100% 61 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 HEAD -m "First stable release" ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 224 in → 70 out (26 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.6:low PASS 100% 62 in → 23 out (0 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
anthropic/claude-opus-4.6:low__json_schema PASS 100% 225 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.6:max PASS 100% 64 in → 72 out (28 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 226 in → 74 out (28 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.6:medium PASS 100% 61 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 HEAD -m "First stable release" ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 226 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.6:none PASS 100% 63 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 HEAD -m "First stable release" ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 225 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.6:xhigh PASS 100% 62 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 HEAD -m "First stable release" ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 225 in → 70 out (26 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:high PASS 100% 83 in → 30 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.7:high__json_schema PASS 100% 295 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:low PASS 100% 85 in → 27 out (0 reasoning)
`git tag -a v1.0.0 -m 'First stable release'`
anthropic/claude-opus-4.7:low__json_schema PASS 100% 294 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:max PASS 100% 84 in → 29 out (0 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 294 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:medium PASS 100% 85 in → 27 out (0 reasoning)
`git tag -a v1.0.0 -m 'First stable release'`
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 294 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:none PASS 100% 83 in → 30 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 294 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.7:xhigh PASS 100% 83 in → 29 out (0 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 294 in → 32 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:high PASS 100% 79 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 289 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:low PASS 100% 79 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 289 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:max PASS 100% 78 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 290 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:medium PASS 100% 79 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 288 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:none PASS 100% 80 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 288 in → 29 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-opus-4.8:xhigh PASS 100% 77 in → 27 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 291 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-sonnet-4.6:high PASS 100% 62 in → 72 out (28 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 224 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-sonnet-4.6:low PASS 100% 61 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 224 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-4.6:max PASS 100% 61 in → 125 out (72 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 227 in → 104 out (54 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-sonnet-4.6:medium PASS 100% 63 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 226 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-4.6:none PASS 100% 61 in → 26 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 224 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 62 in → 72 out (28 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 227 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
anthropic/claude-sonnet-5:high PASS 100% 80 in → 31 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' HEAD ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 290 in → 34 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-5:low PASS 100% 80 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
anthropic/claude-sonnet-5:low__json_schema PASS 100% 288 in → 34 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-5:medium PASS 100% 77 in → 31 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' HEAD ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 290 in → 34 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-5:none PASS 100% 82 in → 31 out (0 reasoning)
``` git tag -a v1.0.0 -m 'First stable release' HEAD ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 291 in → 33 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
anthropic/claude-sonnet-5:xhigh PASS 100% 79 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 288 in → 34 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
arcee-ai/trinity-large-thinking:high PASS 100% 58 in → 225 out (205 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-large-thinking:low PASS 100% 57 in → 337 out (317 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-large-thinking:medium PASS 100% 57 in → 240 out (220 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-large-thinking:xhigh PASS 100% 57 in → 244 out (224 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-mini:high PASS 100% 57 in → 364 out (360 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-mini:high__json_schema PASS 100% 56 in → 431 out (417 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command":"git tag -a v1.0.0 -m \"First stable release\"" }
arcee-ai/trinity-mini:low PASS 100% 59 in → 381 out (369 reasoning)
git tag -a v1.0.0 -m "First stable release"
arcee-ai/trinity-mini:low__json_schema PASS 100% 57 in → 331 out (317 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command":"git tag -a v1.0.0 -m \"First stable release\"" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 56 in → 383 out (366 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 56 in → 324 out (310 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 129 in → 68 out (50 reasoning)
git tag -a v1.0.0 -m 'First stable release'
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 130 in → 69 out (42 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
deepseek/deepseek-v4-flash-0731:low PASS 100% 51 in → 157 out (127 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 51 in → 89 out (61 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 131 in → 90 out (66 reasoning)
git tag -a v1.0.0 -m 'First stable release'
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 131 in → 68 out (34 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 52 in → 64 out (45 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 51 in → 260 out (206 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release' HEAD" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 52 in → 17 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 51 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 51 in → 83 out (59 reasoning)
git tag -a v1.0.0 -m "First stable release"
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 51 in → 143 out (93 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
deepseek/deepseek-v4-flash:high PASS 100% 52 in → 91 out (71 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 53 in → 98 out (70 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash:low PASS 100% 50 in → 180 out (160 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 54 in → 106 out (79 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash:medium PASS 100% 54 in → 94 out (74 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 52 in → 144 out (105 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
deepseek/deepseek-v4-flash:none PASS 100% 52 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 116 in → 24 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
deepseek/deepseek-v4-flash:xhigh PASS 100% 130 in → 249 out (231 reasoning)
git tag -a v1.0.0 -m 'First stable release'
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 131 in → 215 out (186 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
deepseek/deepseek-v4-pro:high PASS 100% 52 in → 91 out (73 reasoning)
git tag -a v1.0.0 -m "First stable release"
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 239 in → 226 out (201 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\" HEAD"}
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 237 in → 319 out (295 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
deepseek/deepseek-v4-pro:medium PASS 100% 52 in → 128 out (109 reasoning)
git tag -a v1.0.0 -m "First stable release"
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 54 in → 113 out (86 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
deepseek/deepseek-v4-pro:none PASS 100% 52 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
deepseek/deepseek-v4-pro:xhigh PASS 100% 133 in → 227 out (208 reasoning)
git tag -a v1.0.0 -m "First stable release"
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 129 in → 216 out (179 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3-flash-preview:high PASS 100% 53 in → 127 out (110 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3-flash-preview:high__json_schema PASS 100% 54 in → 398 out (376 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3-flash-preview:low PASS 100% 53 in → 126 out (109 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemini-3-flash-preview:low__json_schema PASS 100% 110 in → 262 out (240 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3-flash-preview:medium PASS 100% 51 in → 215 out (198 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemini-3-flash-preview:medium__json_schema PASS 100% 110 in → 289 out (267 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3-flash-preview:xhigh PASS 100% 53 in → 144 out (127 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 52 in → 326 out (305 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-flash-lite-preview:high PASS 100% 53 in → 285 out (268 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 109 in → 471 out (449 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-flash-lite-preview:low PASS 100% 54 in → 144 out (126 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 110 in → 161 out (139 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 53 in → 176 out (158 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 52 in → 205 out (183 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 53 in → 362 out (345 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 52 in → 411 out (389 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-pro-preview:high PASS 100% 53 in → 320 out (303 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 107 in → 269 out (247 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-pro-preview:low PASS 100% 52 in → 325 out (308 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 110 in → 200 out (178 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-pro-preview:medium PASS 100% 51 in → 234 out (217 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 108 in → 248 out (227 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.1-pro-preview:xhigh PASS 100% 53 in → 274 out (257 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 109 in → 276 out (254 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash-lite:high PASS 100% 54 in → 420 out (403 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 110 in → 435 out (414 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash-lite:low PASS 100% 53 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 110 in → 28 out (0 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 HEAD -m \"First stable release\"" }
google/gemini-3.5-flash-lite:max PASS 100% 51 in → 402 out (385 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 110 in → 466 out (445 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash-lite:medium PASS 100% 52 in → 219 out (202 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 109 in → 200 out (177 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 HEAD -m \"First stable release\""}
google/gemini-3.5-flash-lite:xhigh PASS 100% 51 in → 252 out (235 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 109 in → 605 out (583 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
google/gemini-3.5-flash:high PASS 100% 51 in → 441 out (424 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash:high__json_schema PASS 100% 109 in → 288 out (266 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash:low PASS 100% 53 in → 344 out (327 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash:low__json_schema PASS 100% 109 in → 135 out (113 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash:medium PASS 100% 52 in → 386 out (369 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash:medium__json_schema PASS 100% 110 in → 364 out (342 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.5-flash:xhigh PASS 100% 52 in → 446 out (429 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 110 in → 316 out (294 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.6-flash:high PASS 100% 53 in → 330 out (313 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.6-flash:high__json_schema PASS 100% 109 in → 369 out (347 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.6-flash:low PASS 100% 52 in → 228 out (211 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.6-flash:low__json_schema PASS 100% 109 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
google/gemini-3.6-flash:max PASS 100% 52 in → 391 out (374 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.6-flash:max__json_schema PASS 100% 110 in → 285 out (258 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
google/gemini-3.6-flash:medium PASS 100% 54 in → 273 out (256 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.6-flash:medium__json_schema PASS 100% 54 in → 150 out (129 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemini-3.6-flash:xhigh PASS 100% 53 in → 256 out (239 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 108 in → 293 out (271 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemma-4-26b-a4b-it:high PASS 100% 69 in → 197 out (143 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 68 in → 231 out (165 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
google/gemma-4-26b-a4b-it:low PASS 100% 69 in → 213 out (154 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 68 in → 239 out (169 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
google/gemma-4-26b-a4b-it:medium PASS 100% 68 in → 242 out (189 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 69 in → 256 out (183 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
google/gemma-4-26b-a4b-it:none PASS 100% 63 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 66 in → 29 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release' HEAD" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 69 in → 164 out (118 reasoning)
git tag -a v1.0.0 -m 'First stable release'
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 69 in → 192 out (134 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
google/gemma-4-31b-it:high PASS 100% 67 in → 145 out (104 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-31b-it:high__json_schema PASS 100% 67 in → 169 out (111 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
google/gemma-4-31b-it:low PASS 100% 67 in → 145 out (102 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-31b-it:low__json_schema PASS 100% 72 in → 152 out (129 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemma-4-31b-it:medium PASS 100% 69 in → 172 out (122 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-31b-it:medium__json_schema PASS 100% 66 in → 182 out (130 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
google/gemma-4-31b-it:none PASS 100% 66 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-31b-it:none__json_schema PASS 100% 66 in → 23 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
google/gemma-4-31b-it:xhigh PASS 100% 67 in → 203 out (149 reasoning)
git tag -a v1.0.0 -m "First stable release"
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 69 in → 147 out (98 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
ibm-granite/granite-4.1-8b PASS 100% 59 in → 18 out
git tag -a v1.0.0 -m "First stable release"
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
inclusionai/ling-2.6-flash PASS 100% 72 in → 24 out
```bash git tag -a v1.0.0 -m "First stable release" HEAD ```
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m 'First stable release'"}
liquid/lfm-2-24b-a2b PASS 100% 59 in → 18 out
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.5:high PASS 100% 86 in → 117 out (98 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.5:high__json_schema PASS 100% 85 in → 111 out (85 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
minimax/minimax-m2.5:low PASS 100% 90 in → 117 out (98 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.5:low__json_schema PASS 100% 85 in → 102 out (79 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
minimax/minimax-m2.5:medium PASS 100% 84 in → 79 out (60 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.5:medium__json_schema PASS 100% 85 in → 110 out (93 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
minimax/minimax-m2.5:xhigh PASS 100% 85 in → 93 out (78 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 84 in → 102 out (77 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
minimax/minimax-m2.7:high PASS 100% 84 in → 314 out (271 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m2.7:high__json_schema PASS 100% 218 in → 802 out (770 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
minimax/minimax-m2.7:low PASS 100% 85 in → 215 out (185 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
minimax/minimax-m2.7:medium PASS 100% 84 in → 175 out (143 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
minimax/minimax-m2.7:medium__json_schema PASS 100% 216 in → 483 out (458 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
minimax/minimax-m2.7:xhigh PASS 100% 86 in → 291 out (273 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m3:high PASS 100% 224 in → 58 out (38 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
minimax/minimax-m3:high__json_schema PASS 100% 173 in → 26 out (25 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
minimax/minimax-m3:low PASS 100% 220 in → 72 out (46 reasoning)
git tag -a v1.0.0 -m 'First stable release'
minimax/minimax-m3:medium PASS 100% 221 in → 91 out (60 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m3:medium__json_schema PASS 100% 210 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
minimax/minimax-m3:xhigh PASS 100% 222 in → 77 out (55 reasoning)
git tag -a v1.0.0 -m "First stable release"
minimax/minimax-m3:xhigh__json_schema PASS 100% 211 in → 26 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release' HEAD" }
mistralai/devstral-2512 PASS 100% 52 in → 24 out
```bash git tag -a v1.0.0 -m "First stable release" HEAD ```
mistralai/devstral-2512__json_schema PASS 100% 52 in → 24 out
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
mistralai/mistral-medium-3-5:high PASS 100% 64 in → 568 out (541 reasoning)
git tag -a v1.0.0 -m "First stable release"
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 53 in → 279 out (243 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
mistralai/mistral-medium-3-5:low PASS 100% 64 in → 232 out (204 reasoning)
git tag -a v1.0.0 -m "First stable release"
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 53 in → 722 out (681 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
mistralai/mistral-medium-3-5:medium PASS 100% 64 in → 293 out (267 reasoning)
git tag -a v1.0.0 -m "First stable release"
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 53 in → 219 out (198 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
mistralai/mistral-medium-3-5:none PASS 100% 64 in → 24 out (0 reasoning)
```bash git tag -a v1.0.0 -m 'First stable release' HEAD ```
mistralai/mistral-medium-3-5:xhigh PASS 100% 63 in → 399 out (378 reasoning)
git tag -a v1.0.0 -m "First stable release"
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 53 in → 245 out (213 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
moonshotai/kimi-k2.6:high PASS 100% 55 in → 148 out (129 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.6:high__json_schema PASS 100% 54 in → 265 out (237 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
moonshotai/kimi-k2.6:low PASS 100% 54 in → 695 out (625 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.6:medium PASS 100% 57 in → 309 out (296 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 55 in → 354 out (200 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
moonshotai/kimi-k2.6:xhigh PASS 100% 55 in → 388 out (369 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 56 in → 25 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
moonshotai/kimi-k2.7-code:high PASS 100% 56 in → 111 out (92 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 137 in → 113 out (88 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\" HEAD"}
moonshotai/kimi-k2.7-code:low PASS 100% 54 in → 111 out (92 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.7-code:medium PASS 100% 56 in → 132 out (113 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 55 in → 94 out (69 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 55 in → 81 out (62 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 137 in → 87 out (63 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
moonshotai/kimi-k3:high PASS 100% 132 in → 117 out (85 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:high__json_schema PASS 100% 227 in → 146 out (110 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
moonshotai/kimi-k3:low PASS 100% 132 in → 108 out (76 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:low__json_schema PASS 100% 132 in → 96 out (64 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:max PASS 100% 131 in → 108 out (76 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:max__json_schema PASS 100% 132 in → 135 out (103 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:medium PASS 100% 133 in → 75 out (43 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:medium__json_schema PASS 100% 131 in → 156 out (124 reasoning)
git tag -a v1.0.0 -m "First stable release"
moonshotai/kimi-k3:xhigh PASS 100% 133 in → 127 out (91 reasoning)
```bash git tag -a v1.0.0 -m "First stable release" ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 133 in → 141 out (109 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 65 in → 130 out (105 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 66 in → 200 out (181 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\"" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 65 in → 94 out (70 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 66 in → 96 out (68 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 65 in → 206 out (199 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 66 in → 115 out (87 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 66 in → 28 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" && git push origin v1.0.0
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 65 in → 26 out (0 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 HEAD -m 'First stable release'" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 66 in → 156 out (121 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 65 in → 79 out (53 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 64 in → 123 out (96 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 66 in → 128 out (107 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 66 in → 93 out (66 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 66 in → 164 out (156 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 65 in → 112 out (85 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 66 in → 19 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 65 in → 24 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 64 in → 109 out (82 reasoning)
git tag -a v1.0.0 -m "First stable release"
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 64 in → 82 out (56 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
openai/gpt-5.4-mini:high PASS 100% 52 in → 98 out (74 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-mini:high__json_schema PASS 100% 87 in → 209 out (177 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m \"First stable release\""}
openai/gpt-5.4-mini:low PASS 100% 52 in → 34 out (10 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
openai/gpt-5.4-mini:low__json_schema PASS 100% 88 in → 42 out (10 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m 'First stable release'"}
openai/gpt-5.4-mini:medium PASS 100% 54 in → 72 out (48 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-mini:medium__json_schema PASS 100% 88 in → 144 out (112 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4-mini:none PASS 100% 55 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-mini:none__json_schema PASS 100% 87 in → 30 out (0 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m 'First stable release'"}
openai/gpt-5.4-mini:xhigh PASS 100% 52 in → 382 out (359 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 90 in → 2,866 out (2,834 reasoning)
git tag -a -m "First stable release" v1.0.0 HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a -m \"First stable release\" v1.0.0 HEAD"}
openai/gpt-5.4-nano:high PASS 100% 52 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-nano:high__json_schema PASS 100% 90 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4-nano:low PASS 100% 52 in → 25 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" 722c926
openai/gpt-5.4-nano:low__json_schema PASS 100% 88 in → 34 out (0 reasoning)
git tag -a v1.0.0 86aeb50 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 86aeb50 -m \"First stable release\""}
openai/gpt-5.4-nano:medium PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-nano:medium__json_schema PASS 100% 87 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4-nano:none PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-nano:none__json_schema PASS 100% 87 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4-nano:xhigh PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 89 in → 35 out (0 reasoning)
git tag -a v1.0.0 0c396f4 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 0c396f4 -m \"First stable release\""}
openai/gpt-5.4:high PASS 100% 52 in → 84 out (60 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.4:high__json_schema PASS 100% 89 in → 145 out (113 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4:low PASS 100% 52 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4:low__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4:medium PASS 100% 55 in → 85 out (61 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4:medium__json_schema PASS 100% 90 in → 88 out (56 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.4:none PASS 100% 53 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.4:none__json_schema PASS 100% 89 in → 29 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
openai/gpt-5.4:xhigh PASS 100% 54 in → 230 out (206 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.4:xhigh__json_schema PASS 100% 89 in → 380 out (348 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
openai/gpt-5.5:high PASS 100% 53 in → 77 out (53 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.5:high__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.5:low PASS 100% 53 in → 21 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-5.5:low__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.5:medium PASS 100% 53 in → 67 out (43 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.5:medium__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.5:none PASS 100% 55 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.5:none__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.5:xhigh PASS 100% 53 in → 104 out (80 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.5:xhigh__json_schema PASS 100% 89 in → 184 out (151 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-luna:high PASS 100% 54 in → 54 out (30 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-luna:high__json_schema PASS 100% 89 in → 69 out (37 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
openai/gpt-5.6-luna:low PASS 100% 54 in → 62 out (38 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.6-luna:low__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-luna:max PASS 100% 52 in → 103 out (79 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-luna:max__json_schema PASS 100% 88 in → 472 out (440 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-luna:medium PASS 100% 55 in → 67 out (43 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-luna:medium__json_schema PASS 100% 87 in → 62 out (30 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
openai/gpt-5.6-luna:none PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-luna:none__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-luna:xhigh PASS 100% 54 in → 76 out (52 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 87 in → 261 out (229 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
openai/gpt-5.6-sol:high PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.6-sol:high__json_schema PASS 100% 90 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-sol:low PASS 100% 53 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-sol:low__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-sol:max PASS 100% 55 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-sol:max__json_schema PASS 100% 89 in → 145 out (113 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
openai/gpt-5.6-sol:medium PASS 100% 53 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-sol:medium__json_schema PASS 100% 89 in → 29 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
openai/gpt-5.6-sol:none PASS 100% 53 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-sol:none__json_schema PASS 100% 87 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-sol:xhigh PASS 100% 53 in → 21 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:high PASS 100% 52 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-terra:high__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:low PASS 100% 54 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-terra:low__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:max PASS 100% 55 in → 68 out (44 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
openai/gpt-5.6-terra:max__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:medium PASS 100% 53 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-terra:medium__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:none PASS 100% 55 in → 22 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-terra:none__json_schema PASS 100% 88 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-5.6-terra:xhigh PASS 100% 55 in → 60 out (36 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 89 in → 30 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-oss-120b:high PASS 100% 117 in → 375 out (340 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-120b:high__json_schema PASS 100% 113 in → 25 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
openai/gpt-oss-120b:low PASS 100% 103 in → 59 out (29 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-120b:low__json_schema PASS 100% 112 in → 59 out (20 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
openai/gpt-oss-120b:medium PASS 100% 124 in → 95 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-120b:medium__json_schema PASS 100% 113 in → 93 out (61 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
openai/gpt-oss-120b:xhigh PASS 100% 104 in → 216 out (185 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 113 in → 257 out (235 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
openai/gpt-oss-20b:high PASS 100% 115 in → 277 out (252 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-oss-20b:high__json_schema PASS 100% 111 in → 315 out (255 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
openai/gpt-oss-20b:low PASS 100% 101 in → 74 out (40 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
openai/gpt-oss-20b:low__json_schema PASS 100% 102 in → 57 out (15 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
openai/gpt-oss-20b:medium PASS 100% 113 in → 80 out (50 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-20b:medium__json_schema PASS 100% 115 in → 227 out (195 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
openai/gpt-oss-20b:xhigh PASS 100% 115 in → 94 out (57 reasoning)
git tag -a v1.0.0 -m "First stable release"
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 114 in → 310 out (284 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command":"git tag -a v1.0.0 -m \"First stable release\" HEAD"}
poolside/laguna-m.1:high PASS 100% 67 in → 194 out (174 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-m.1:high__json_schema PASS 100% 65 in → 363 out (340 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
poolside/laguna-m.1:low PASS 100% 67 in → 331 out (313 reasoning)
git tag -a v1.0.0 -m "First stable release"
poolside/laguna-m.1:low__json_schema PASS 100% 66 in → 1,073 out (1,051 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
poolside/laguna-m.1:medium PASS 100% 67 in → 300 out (280 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-m.1:medium__json_schema PASS 100% 67 in → 347 out (325 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 HEAD -m \"First stable release\""}
poolside/laguna-m.1:none PASS 100% 65 in → 19 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-m.1:xhigh PASS 100% 67 in → 594 out (576 reasoning)
git tag -a v1.0.0 -m "First stable release"
poolside/laguna-m.1:xhigh__json_schema PASS 100% 65 in → 195 out (169 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
poolside/laguna-xs-2.1:high PASS 100% 64 in → 438 out (423 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs-2.1:high__json_schema PASS 100% 67 in → 305 out (279 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
poolside/laguna-xs-2.1:low PASS 100% 67 in → 233 out (217 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs-2.1:low__json_schema PASS 100% 67 in → 339 out (314 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
poolside/laguna-xs-2.1:medium PASS 100% 66 in → 338 out (322 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 67 in → 281 out (255 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
poolside/laguna-xs-2.1:none PASS 100% 67 in → 17 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs-2.1:none__json_schema PASS 100% 65 in → 27 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
poolside/laguna-xs-2.1:xhigh PASS 100% 65 in → 271 out (255 reasoning)
git tag -a v1.0.0 -m "First stable release"
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 66 in → 286 out (264 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
poolside/laguna-xs.2:high PASS 100% 104 in → 220 out (200 reasoning)
git tag -a v1.0.0 -m "First stable release"
poolside/laguna-xs.2:high__json_schema PASS 100% 102 in → 319 out (297 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
poolside/laguna-xs.2:low PASS 100% 103 in → 162 out (144 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs.2:low__json_schema PASS 100% 103 in → 166 out (144 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
poolside/laguna-xs.2:medium PASS 100% 102 in → 161 out (141 reasoning)
git tag -a v1.0.0 -m 'First stable release'
poolside/laguna-xs.2:medium__json_schema PASS 100% 104 in → 180 out (158 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
poolside/laguna-xs.2:none PASS 100% 102 in → 20 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
poolside/laguna-xs.2:none__json_schema PASS 100% 104 in → 24 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
poolside/laguna-xs.2:xhigh PASS 100% 103 in → 198 out (177 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 102 in → 160 out (136 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
qwen/qwen3.6-27b:high PASS 100% 61 in → 1,860 out (1,577 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.6-27b:high__json_schema PASS 100% 62 in → 1,314 out (1,154 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
qwen/qwen3.6-27b:low PASS 100% 62 in → 1,366 out (1,209 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
qwen/qwen3.6-27b:low__json_schema PASS 100% 65 in → 20 out (0 reasoning)
git tag v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag v1.0.0 -m 'First stable release'"}
qwen/qwen3.6-27b:medium PASS 100% 63 in → 1,213 out (1,191 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.6-27b:none PASS 100% 64 in → 19 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
qwen/qwen3.6-27b:xhigh PASS 100% 63 in → 4,744 out (4,231 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.6-35b-a3b:high PASS 100% 61 in → 850 out (760 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 63 in → 1,189 out (1,060 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
qwen/qwen3.6-35b-a3b:low PASS 100% 63 in → 1,633 out (1,347 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 62 in → 1,696 out (1,409 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 HEAD -m 'First stable release'" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 62 in → 777 out (754 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 63 in → 1,096 out (952 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release' HEAD" }
qwen/qwen3.6-35b-a3b:none PASS 100% 64 in → 18 out (0 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 63 in → 1,646 out (1,415 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 63 in → 1,178 out (1,026 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
qwen/qwen3.6-flash:high PASS 100% 61 in → 899 out (876 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.6-flash:low PASS 100% 62 in → 1,199 out (1,176 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
qwen/qwen3.6-flash:medium PASS 100% 62 in → 845 out (822 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
qwen/qwen3.6-flash:none PASS 100% 65 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.6-flash:xhigh PASS 100% 61 in → 1,065 out (1,042 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
qwen/qwen3.7-flash:high PASS 100% 63 in → 1,012 out (989 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.7-flash:low PASS 100% 61 in → 788 out (766 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-flash:max PASS 100% 61 in → 825 out (802 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.7-flash:medium PASS 100% 61 in → 780 out (757 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.7-flash:none PASS 100% 64 in → 18 out (0 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
qwen/qwen3.7-flash:xhigh PASS 100% 63 in → 743 out (720 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
qwen/qwen3.7-max:high PASS 100% 62 in → 828 out (806 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-max:low PASS 100% 63 in → 1,096 out (1,074 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-max:medium PASS 100% 60 in → 695 out (673 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-max:none PASS 100% 64 in → 17 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-max:xhigh PASS 100% 62 in → 675 out (652 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.7-plus:high PASS 100% 63 in → 629 out (606 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
qwen/qwen3.7-plus:low PASS 100% 62 in → 963 out (941 reasoning)
git tag -a v1.0.0 -m 'First stable release'
qwen/qwen3.7-plus:medium PASS 100% 62 in → 1,035 out (1,013 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-plus:none PASS 100% 63 in → 17 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
qwen/qwen3.7-plus:xhigh PASS 100% 63 in → 824 out (801 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
tencent/hy3:high PASS 100% 58 in → 479 out (460 reasoning)
git tag -a v1.0.0 -m "First stable release"
tencent/hy3:high__json_schema PASS 100% 62 in → 247 out (228 reasoning)
git tag -a v1.0.0 -m 'First stable release'
tencent/hy3:low PASS 100% 62 in → 348 out (292 reasoning)
git tag -a v1.0.0 -m "First stable release"
tencent/hy3:low__json_schema PASS 100% 60 in → 345 out (313 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
tencent/hy3:medium PASS 100% 60 in → 146 out (127 reasoning)
git tag -a v1.0.0 -m "First stable release"
tencent/hy3:medium__json_schema PASS 100% 60 in → 133 out (95 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
tencent/hy3:none PASS 100% 63 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
tencent/hy3:xhigh PASS 100% 62 in → 360 out (341 reasoning)
git tag -a v1.0.0 -m "First stable release"
tencent/hy3:xhigh__json_schema PASS 100% 61 in → 371 out (336 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m \"First stable release\""}
thinkingmachines/inkling-small:high PASS 100% 62 in → 116 out (85 reasoning)
git tag -a v1.0.0 -m 'First stable release'
thinkingmachines/inkling-small:high__json_schema PASS 100% 62 in → 177 out (149 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\" HEAD"}
thinkingmachines/inkling-small:low PASS 100% 64 in → 47 out (17 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
thinkingmachines/inkling-small:low__json_schema PASS 100% 63 in → 64 out (36 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release' HEAD"}
thinkingmachines/inkling-small:max PASS 100% 62 in → 242 out (218 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
thinkingmachines/inkling-small:max__json_schema PASS 100% 62 in → 177 out (149 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
thinkingmachines/inkling-small:medium PASS 100% 62 in → 128 out (95 reasoning)
git tag -a v1.0.0 -m 'First stable release'
thinkingmachines/inkling-small:medium__json_schema PASS 100% 62 in → 98 out (67 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release' HEAD" }
thinkingmachines/inkling-small:none PASS 100% 61 in → 21 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
thinkingmachines/inkling-small:xhigh PASS 100% 63 in → 208 out (151 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 62 in → 175 out (144 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 HEAD -m \"First stable release\"" }
thinkingmachines/inkling:high PASS 100% 62 in → 118 out (94 reasoning)
git tag -a v1.0.0 -m "First stable release"
thinkingmachines/inkling:low PASS 100% 63 in → 45 out (17 reasoning)
git tag -a v1.0.0 -m "First stable release"
thinkingmachines/inkling:low__json_schema PASS 100% 63 in → 49 out (22 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
thinkingmachines/inkling:max PASS 100% 61 in → 100 out (77 reasoning)
git tag -a v1.0.0 -m "First stable release"
thinkingmachines/inkling:medium PASS 100% 62 in → 99 out (76 reasoning)
git tag -a v1.0.0 -m 'First stable release'
thinkingmachines/inkling:none__json_schema PASS 100% 61 in → 25 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
thinkingmachines/inkling:xhigh PASS 100% 62 in → 284 out (260 reasoning)
git tag -a v1.0.0 HEAD -m "First stable release"
thinkingmachines/inkling:xhigh__json_schema PASS 100% 61 in → 87 out (58 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
x-ai/grok-4.3:high PASS 100% 231 in → 357 out (340 reasoning)
git tag -a v1.0.0 -m "First stable release"
x-ai/grok-4.3:high__json_schema PASS 100% 289 in → 932 out (910 reasoning)
git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release' HEAD"}
x-ai/grok-4.3:low PASS 100% 237 in → 222 out (205 reasoning)
git tag -a v1.0.0 -m "First stable release"
x-ai/grok-4.3:low__json_schema PASS 100% 294 in → 369 out (345 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
x-ai/grok-4.3:max PASS 100% 234 in → 283 out (266 reasoning)
git tag -a v1.0.0 -m "First stable release"
x-ai/grok-4.3:max__json_schema PASS 100% 288 in → 395 out (374 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
x-ai/grok-4.3:medium PASS 100% 241 in → 1,164 out (1,146 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
x-ai/grok-4.3:medium__json_schema PASS 100% 294 in → 501 out (477 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
x-ai/grok-4.3:none PASS 100% 229 in → 20 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" dab4227
x-ai/grok-4.3:none__json_schema PASS 100% 286 in → 22 out (0 reasoning)
git tag -a v1.0.0 HEAD -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 HEAD -m 'First stable release'"}
x-ai/grok-4.3:xhigh PASS 100% 234 in → 455 out (438 reasoning)
git tag -a v1.0.0 -m 'First stable release'
x-ai/grok-4.3:xhigh__json_schema PASS 100% 290 in → 503 out (482 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
x-ai/grok-4.5:high PASS 100% 253 in → 149 out (132 reasoning)
git tag -a v1.0.0 -m "First stable release"
x-ai/grok-4.5:high__json_schema PASS 100% 319 in → 404 out (382 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
x-ai/grok-4.5:low PASS 100% 253 in → 66 out (46 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
x-ai/grok-4.5:low__json_schema PASS 100% 320 in → 315 out (293 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
x-ai/grok-4.5:max PASS 100% 253 in → 164 out (144 reasoning)
``` git tag -a v1.0.0 -m "First stable release" ```
x-ai/grok-4.5:max__json_schema PASS 100% 320 in → 226 out (203 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
x-ai/grok-4.5:medium PASS 100% 253 in → 183 out (166 reasoning)
git tag -a v1.0.0 -m 'First stable release'
x-ai/grok-4.5:medium__json_schema PASS 100% 319 in → 239 out (216 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m \"First stable release\""}
x-ai/grok-4.5:xhigh PASS 100% 255 in → 154 out (137 reasoning)
git tag -a v1.0.0 -m "First stable release"
x-ai/grok-4.5:xhigh__json_schema PASS 100% 320 in → 250 out (228 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command":"git tag -a v1.0.0 -m 'First stable release'"}
z-ai/glm-4.7-flash:high PASS 100% 52 in → 365 out (301 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-4.7-flash:high__json_schema PASS 100% 53 in → 411 out (338 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
z-ai/glm-4.7-flash:low PASS 100% 53 in → 316 out (271 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-4.7-flash:low__json_schema PASS 100% 53 in → 455 out (394 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
z-ai/glm-4.7-flash:medium PASS 100% 54 in → 577 out (484 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 53 in → 406 out (354 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
z-ai/glm-4.7-flash:none PASS 100% 52 in → 19 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release" HEAD
z-ai/glm-4.7-flash:none__json_schema PASS 100% 53 in → 23 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
z-ai/glm-4.7-flash:xhigh PASS 100% 52 in → 379 out (328 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 53 in → 290 out (236 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m \"First stable release\""}
z-ai/glm-5.2:high PASS 100% 60 in → 222 out (203 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-5.2:high__json_schema PASS 100% 62 in → 300 out (275 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
z-ai/glm-5.2:low PASS 100% 60 in → 249 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-5.2:low__json_schema PASS 100% 59 in → 241 out (203 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m 'First stable release'" }
z-ai/glm-5.2:medium PASS 100% 60 in → 313 out (276 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-5.2:medium__json_schema PASS 100% 60 in → 376 out (335 reasoning)
git tag -a v1.0.0 -m 'First stable release'
JSON Schema Structured Output
(raw) {"command": "git tag -a v1.0.0 -m 'First stable release'"}
z-ai/glm-5.2:none PASS 100% 54 in → 18 out (0 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-5.2:xhigh PASS 100% 59 in → 596 out (487 reasoning)
git tag -a v1.0.0 -m "First stable release"
z-ai/glm-5.2:xhigh__json_schema PASS 100% 60 in → 429 out (374 reasoning)
git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
(raw) { "command": "git tag -a v1.0.0 -m \"First stable release\"" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:medium FAIL 0% 57 in → 335 out (329 reasoning)
git tag -av1.0.0 -m "First stable release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0.0'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0.0', 'contains': 'First stable release'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0.0', 'contains': 'tag'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 56 in → 378 out (369 reasoning)
gittag -a v1.0.0 -m "First stable release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0.0'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0.0', 'contains': 'First stable release'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0.0', 'contains': 'tag'}]
deepseek/deepseek-v4-pro:low FAIL 0% 51 in → 94 out (69 reasoning)
git tag -a v1.0.0-m "First stable release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0.0'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0.0', 'contains': 'First stable release'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0.0', 'contains': 'tag'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 65 in → 196 out (172 reasoning)
gittag -a v1.0.0 -m "First stable release"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0.0'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0.0', 'contains': 'First stable release'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0.0', 'contains': 'tag'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: git tag -a v1.0.0 -m 'First stable release' HEAD
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-355620f0-dcb4-9485-81dc-7433e4f53c05","request_id":"355620f0-dcb4-9485-81dc-7433e4f53c05"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-641ef3ed-de8d-9b21-b2fd-0f595eec5ede","request_id":"641ef3ed-de8d-9b21-b2fd-0f595eec5ede"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5fa6b9b0-c888-99e2-be41-8c245c0ff60f","request_id":"5fa6b9b0-c888-99e2-be41-8c245c0ff60f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c529d965-e2de-916b-bff6-44cd4ee44746","request_id":"c529d965-e2de-916b-bff6-44cd4ee44746"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-24a6446e-19d4-9ea7-9b98-b32af598f7b6","request_id":"24a6446e-19d4-9ea7-9b98-b32af598f7b6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0f806a0c-572d-9e72-a41f-5110663791af"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0570c6aa-e649-9106-83b1-f2917d5fef45"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-88efbbb5-b085-95d6-8e48-a9b64e4c00d8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-367ca05b-0062-9319-b417-8282bd0bf9cf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ee3b8ce3-41b6-90f1-af99-b43fbd846957"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f9a7a7dc-3ffd-9395-95c8-8467d75379b5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3c4f312c-a1a1-9ad4-8912-c3895961ece7","request_id":"3c4f312c-a1a1-9ad4-8912-c3895961ece7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7fd2dc6b-5700-9f61-ad31-f6e05e63c204","request_id":"7fd2dc6b-5700-9f61-ad31-f6e05e63c204"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d7c95bf5-3c75-9770-8df5-43504a42e3a2","request_id":"d7c95bf5-3c75-9770-8df5-43504a42e3a2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4407d040-e53b-9c02-82ab-3c209c904f73","request_id":"4407d040-e53b-9c02-82ab-3c209c904f73"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f4ca7e81-2c4c-9244-a85b-58cf3202cad1","request_id":"f4ca7e81-2c4c-9244-a85b-58cf3202cad1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3b2708cf-60fc-9728-8ee7-40397e5ae70f","request_id":"3b2708cf-60fc-9728-8ee7-40397e5ae70f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f4462aac-f5fc-920a-bdb5-4354fe32e1af","request_id":"f4462aac-f5fc-920a-bdb5-4354fe32e1af"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ba37b395-e385-9dbd-8fde-68616c1d3874","request_id":"ba37b395-e385-9dbd-8fde-68616c1d3874"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Model 'tencent/hy3:none' violated the reasoning_level=none invariant: provider reported 808 reasoning token(s)
Invalid JSON. Output: git tag -a v1.0.0 -m "First stable release" HEAD{ "command": "git tag -a v1.0.0 -m \"First stable release\" HEAD" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag -a v1.0.0 -m "First stable release"
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 63 in → 20 out (0 reasoning)
git tag -a v1.0.0 -m 'First stable release'<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v1.0.0'}, {'type': 'git_output', 'command': 'git tag -n1 v1.0.0', 'contains': 'First stable release'}, {'type': 'git_output', 'command': 'git cat-file -t v1.0.0', 'contains': 'tag'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.