Tag an older commit
Tests ability to tag an older (non-HEAD) commit. Evaluates retroactive-tagging command knowledge.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'v1' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'First commit'
  7. 07 echo 'v2' > app.txt
  8. 08 git add app.txt
  9. 09 git commit -m 'Second commit'
  10. 10 echo 'v3' > app.txt
  11. 11 git add app.txt
  12. 12 git commit -m 'Third commit'
Prompt
Tag the commit with message 'First commit' as 'v0.1'. Output ONLY the git command, nothing else.
Expected
git tag v0.1 HEAD~2

Scoped model quality, cost, API time, and token usage for tag_management/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 87 in → 30 out (16 reasoning)
``` git tag v0.1 054f301 ```
anthropic/claude-fable-5:high__json_schema PASS 100% 298 in → 21 out (0 reasoning)
git tag v0.1 e116a58
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 e116a58"}
anthropic/claude-fable-5:low PASS 100% 93 in → 19 out (0 reasoning)
``` git tag v0.1 dee16b2 ```
anthropic/claude-fable-5:low__json_schema PASS 100% 297 in → 20 out (0 reasoning)
git tag v0.1 225f326
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 225f326"}
anthropic/claude-fable-5:max PASS 100% 90 in → 91 out (34 reasoning)
git tag v0.1 c6ea633
anthropic/claude-fable-5:max__json_schema PASS 100% 297 in → 115 out (33 reasoning)
git tag v0.1 48ce869
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 48ce869"}
anthropic/claude-fable-5:medium PASS 100% 88 in → 19 out (0 reasoning)
``` git tag v0.1 ef87e25 ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 299 in → 22 out (0 reasoning)
git tag v0.1 fd7c399
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 fd7c399"}
anthropic/claude-fable-5:xhigh PASS 100% 83 in → 47 out (24 reasoning)
git tag v0.1 da076de
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 299 in → 40 out (19 reasoning)
git tag v0.1 e9734ac
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 e9734ac"}
anthropic/claude-haiku-4.5:high PASS 100% 100 in → 142 out (101 reasoning)
``` git tag v0.1 d0d1855 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 263 in → 217 out (143 reasoning)
git tag v0.1 9f848b2
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 9f848b2"}
anthropic/claude-haiku-4.5:low PASS 100% 98 in → 158 out (111 reasoning)
git tag v0.1 ba63c6f
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 263 in → 199 out (141 reasoning)
git tag v0.1 7bb3ff9
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 7bb3ff9"}
anthropic/claude-haiku-4.5:medium PASS 100% 99 in → 159 out (107 reasoning)
``` git tag v0.1 1cc0357 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 262 in → 277 out (194 reasoning)
git tag v0.1 80cae7d
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 80cae7d"}
anthropic/claude-haiku-4.5:none PASS 100% 68 in → 18 out (0 reasoning)
``` git tag v0.1 f9178af ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 232 in → 29 out (0 reasoning)
git tag -a v0.1 8defc52 -m 'First commit'
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 8defc52 -m 'First commit'"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 100 in → 179 out (119 reasoning)
``` git tag v0.1 ddbe7b7 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 261 in → 198 out (159 reasoning)
git tag v0.1 bd610d8
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 bd610d8"}
anthropic/claude-opus-4.6:high PASS 100% 72 in → 66 out (24 reasoning)
git tag v0.1 7e47a8f
anthropic/claude-opus-4.6:high__json_schema PASS 100% 231 in → 71 out (29 reasoning)
git tag v0.1 2cdff71
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 2cdff71"}
anthropic/claude-opus-4.6:low PASS 100% 71 in → 15 out (0 reasoning)
git tag v0.1 4aeeee5
anthropic/claude-opus-4.6:low__json_schema PASS 100% 228 in → 17 out (0 reasoning)
git tag v0.1 062a560
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 062a560"}
anthropic/claude-opus-4.6:max PASS 100% 70 in → 74 out (33 reasoning)
git tag v0.1 5b1ab53
anthropic/claude-opus-4.6:max__json_schema PASS 100% 230 in → 100 out (55 reasoning)
git tag v0.1 37f5140
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 37f5140"}
anthropic/claude-opus-4.6:medium PASS 100% 70 in → 16 out (0 reasoning)
git tag v0.1 62b8a8c
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 233 in → 18 out (0 reasoning)
git tag v0.1 c07219e
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 c07219e"}
anthropic/claude-opus-4.6:none PASS 100% 69 in → 19 out (0 reasoning)
``` git tag v0.1 f9510a1 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 233 in → 19 out (0 reasoning)
git tag v0.1 ed62c8e
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 ed62c8e"}
anthropic/claude-opus-4.6:xhigh PASS 100% 69 in → 68 out (29 reasoning)
git tag v0.1 6629e0e
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 233 in → 66 out (22 reasoning)
git tag v0.1 68c56d1
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 68c56d1"}
anthropic/claude-opus-4.7:high PASS 100% 91 in → 20 out (0 reasoning)
``` git tag v0.1 885c416 ```
anthropic/claude-opus-4.7:high__json_schema PASS 100% 302 in → 25 out (0 reasoning)
git tag v0.1 50dbc8f
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 50dbc8f"}
anthropic/claude-opus-4.7:low PASS 100% 88 in → 18 out (0 reasoning)
`git tag v0.1 f28328a`
anthropic/claude-opus-4.7:low__json_schema PASS 100% 303 in → 24 out (0 reasoning)
git tag v0.1 d0d1447
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d0d1447"}
anthropic/claude-opus-4.7:max PASS 100% 94 in → 22 out (0 reasoning)
``` git tag v0.1 28fe03a ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 306 in → 25 out (0 reasoning)
git tag v0.1 c0c41e8
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 c0c41e8"}
anthropic/claude-opus-4.7:medium PASS 100% 95 in → 21 out (0 reasoning)
`git tag v0.1 3d9f8cb`
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 302 in → 24 out (0 reasoning)
git tag v0.1 d5a67df
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d5a67df"}
anthropic/claude-opus-4.7:none PASS 100% 91 in → 21 out (0 reasoning)
``` git tag v0.1 20904b1 ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 301 in → 24 out (0 reasoning)
git tag v0.1 d0ec346
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d0ec346"}
anthropic/claude-opus-4.7:xhigh PASS 100% 95 in → 24 out (0 reasoning)
``` git tag v0.1 5e2a9e8 ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 305 in → 24 out (0 reasoning)
git tag v0.1 328f6e1
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 328f6e1"}
anthropic/claude-opus-4.8:high PASS 100% 87 in → 19 out (0 reasoning)
``` git tag v0.1 99b60ab ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 299 in → 22 out (0 reasoning)
git tag v0.1 89cfd69
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 89cfd69"}
anthropic/claude-opus-4.8:low PASS 100% 86 in → 19 out (0 reasoning)
``` git tag v0.1 9a814e4 ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 297 in → 22 out (0 reasoning)
git tag v0.1 9915ecb
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 9915ecb"}
anthropic/claude-opus-4.8:max PASS 100% 90 in → 167 out (38 reasoning)
`git tag v0.1 c82e8c1`
anthropic/claude-opus-4.8:max__json_schema PASS 100% 298 in → 78 out (22 reasoning)
git tag v0.1 7221d3e
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 7221d3e"}
anthropic/claude-opus-4.8:medium PASS 100% 87 in → 19 out (0 reasoning)
``` git tag v0.1 f98eb79 ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 298 in → 21 out (0 reasoning)
git tag v0.1 78372c6
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 78372c6"}
anthropic/claude-opus-4.8:none PASS 100% 88 in → 19 out (0 reasoning)
``` git tag v0.1 cd695eb ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 298 in → 22 out (0 reasoning)
git tag v0.1 3d02ae9
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 3d02ae9"}
anthropic/claude-opus-4.8:xhigh PASS 100% 86 in → 15 out (0 reasoning)
git tag v0.1 2eb0871
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 296 in → 21 out (0 reasoning)
git tag v0.1 fc48812
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 fc48812"}
anthropic/claude-sonnet-4.6:high PASS 100% 71 in → 77 out (33 reasoning)
``` git tag v0.1 56eb6a8 ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 233 in → 74 out (30 reasoning)
git tag v0.1 e46b5f6
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 e46b5f6"}
anthropic/claude-sonnet-4.6:low PASS 100% 70 in → 19 out (0 reasoning)
``` git tag v0.1 11a1de9 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 232 in → 19 out (0 reasoning)
git tag v0.1 1d7420d
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 1d7420d"}
anthropic/claude-sonnet-4.6:max PASS 100% 66 in → 98 out (56 reasoning)
``` git tag v0.1 a657814 ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 232 in → 95 out (47 reasoning)
git tag v0.1 7b9278c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 7b9278c"}
anthropic/claude-sonnet-4.6:medium PASS 100% 73 in → 18 out (0 reasoning)
``` git tag v0.1 36914b1 ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 232 in → 18 out (0 reasoning)
git tag v0.1 522aee7
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 522aee7"}
anthropic/claude-sonnet-4.6:none PASS 100% 69 in → 20 out (0 reasoning)
``` git tag v0.1 e0ef8b8 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 232 in → 18 out (0 reasoning)
git tag v0.1 988cd54
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 988cd54"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 69 in → 74 out (33 reasoning)
``` git tag v0.1 b1bc252 ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 232 in → 72 out (30 reasoning)
git tag v0.1 026eeb6
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 026eeb6"}
anthropic/claude-sonnet-5:high PASS 100% 88 in → 16 out (0 reasoning)
`git tag v0.1 f778c83`
anthropic/claude-sonnet-5:high__json_schema PASS 100% 294 in → 20 out (0 reasoning)
git tag v0.1 a540466
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 a540466"}
anthropic/claude-sonnet-5:low PASS 100% 88 in → 19 out (0 reasoning)
``` git tag v0.1 82d8f31 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 295 in → 20 out (0 reasoning)
git tag v0.1 389f776
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 389f776"}
anthropic/claude-sonnet-5:medium PASS 100% 88 in → 20 out (0 reasoning)
``` git tag v0.1 09b5f3e ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 296 in → 21 out (0 reasoning)
git tag v0.1 262bb61
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 262bb61"}
anthropic/claude-sonnet-5:none PASS 100% 88 in → 18 out (0 reasoning)
``` git tag v0.1 7e91721 ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 296 in → 20 out (0 reasoning)
git tag v0.1 b02b331
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 b02b331"}
anthropic/claude-sonnet-5:xhigh PASS 100% 90 in → 20 out (0 reasoning)
``` git tag v0.1 b4e1d11 ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 296 in → 22 out (0 reasoning)
git tag v0.1 8fdc561
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 8fdc561"}
arcee-ai/trinity-large-thinking:high PASS 100% 64 in → 274 out (263 reasoning)
git tag v0.1 b93043e
arcee-ai/trinity-large-thinking:low PASS 100% 65 in → 645 out (635 reasoning)
git tag v0.1 f54e3ca
arcee-ai/trinity-large-thinking:medium PASS 100% 66 in → 140 out (126 reasoning)
git tag v0.1 47a8d79
arcee-ai/trinity-mini:high PASS 100% 67 in → 274 out (236 reasoning)
git tag -a v0.1 d3523d3 -m "First commit"
arcee-ai/trinity-mini:high__json_schema PASS 100% 62 in → 504 out (430 reasoning)
git tag -a v0.1 -m "v0.1" f53187e
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 -m \"v0.1\" f53187e" }
arcee-ai/trinity-mini:low PASS 100% 62 in → 1,653 out (1,589 reasoning)
git tag -a v0.1 00dd414 -m "First commit"
arcee-ai/trinity-mini:medium PASS 100% 62 in → 719 out (677 reasoning)
git tag v0.1 128c940
arcee-ai/trinity-mini:medium__json_schema PASS 100% 64 in → 440 out (406 reasoning)
git tag v0.1 b934ae3
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 b934ae3" }
arcee-ai/trinity-mini:xhigh PASS 100% 62 in → 293 out (261 reasoning)
git tag -a v0.1 f77e390 -m "First commit"
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 66 in → 647 out (579 reasoning)
git tag -a v0.1 -m "First commit" 2ea27d7
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 -m \"First commit\" 2ea27d7" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 139 in → 188 out (148 reasoning)
git tag -a v0.1 e7ae82d -m "First commit"
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 59 in → 529 out (453 reasoning)
git tag -a v0.1 ec69ca3 -m "First commit"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 ec69ca3 -m \"First commit\""}
deepseek/deepseek-v4-flash-0731:low PASS 100% 57 in → 111 out (83 reasoning)
git tag -a v0.1 -m 'First commit' ceeb39d
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 61 in → 81 out (56 reasoning)
git tag -a v0.1 -m 'First commit' e3b491c
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 -m 'First commit' e3b491c"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 140 in → 86 out (64 reasoning)
git tag -a v0.1 d27a9c2 -m "First commit"
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 139 in → 159 out (118 reasoning)
git tag -a v0.1 d82c7e1 -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 d82c7e1 -m 'First commit'"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 58 in → 189 out (144 reasoning)
git tag -a v0.1 -m "First commit" d120a03
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 56 in → 419 out (336 reasoning)
git tag -a v0.1 -m "First commit" 5ae40f6
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 -m \"First commit\" 5ae40f6"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 60 in → 12 out (0 reasoning)
git tag v0.1 4730f61
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 55 in → 28 out (0 reasoning)
git tag -a v0.1 84be5bf -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 84be5bf -m \"First commit\"" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 136 in → 93 out (68 reasoning)
git tag -a v0.1 a631bbd -m "First commit"
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 135 in → 181 out (141 reasoning)
git tag -a v0.1 -m 'First commit' e57afdd
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 -m 'First commit' e57afdd"}
deepseek/deepseek-v4-flash:high PASS 100% 57 in → 82 out (58 reasoning)
git tag v0.1 a47f84e
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 56 in → 906 out (887 reasoning)
git tag v0.1 27945ee
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 27945ee"}
deepseek/deepseek-v4-flash:low PASS 100% 59 in → 100 out (79 reasoning)
git tag -a v0.1 6433eb9 -m "First commit"
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 57 in → 343 out (323 reasoning)
git tag v0.1 fedbe4d
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 fedbe4d" }
deepseek/deepseek-v4-flash:medium PASS 100% 59 in → 100 out (88 reasoning)
git tag v0.1 2f056e6
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 58 in → 373 out (356 reasoning)
git tag v0.1 4407714
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 4407714"}
deepseek/deepseek-v4-flash:none PASS 100% 60 in → 20 out (0 reasoning)
git tag -a v0.1 -m "First commit" b6da21f
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 60 in → 26 out (0 reasoning)
git tag -a v0.1 5755142 -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 5755142 -m 'First commit'" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 135 in → 616 out (562 reasoning)
git tag v0.1 2ce09e2
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 149 in → 25 out (0 reasoning)
git tag -a v0.1 505aef4 -m 'First commit'
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 505aef4 -m 'First commit'"}
deepseek/deepseek-v4-pro:high PASS 100% 59 in → 20 out (0 reasoning)
git tag -a v0.1 8ea4226 -m "First commit"
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 246 in → 109 out (91 reasoning)
git tag v0.1 02b66b6
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 02b66b6"}
deepseek/deepseek-v4-pro:low PASS 100% 55 in → 19 out (0 reasoning)
git tag -a v0.1 cdb9476 -m "First commit"
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 61 in → 27 out (0 reasoning)
git tag v0.1 0174a8e -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 0174a8e -m \"First commit\"" }
deepseek/deepseek-v4-pro:medium PASS 100% 57 in → 17 out (0 reasoning)
git tag v0.1 cc2030f -m "First commit"
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 246 in → 212 out (194 reasoning)
git tag v0.1 9631d30
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 9631d30"}
deepseek/deepseek-v4-pro:none PASS 100% 56 in → 10 out (0 reasoning)
git tag v0.1 415fd21
deepseek/deepseek-v4-pro:xhigh PASS 100% 135 in → 491 out (480 reasoning)
git tag v0.1 aa991bd
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 60 in → 27 out (0 reasoning)
git tag -a v0.1 b20f289 -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 b20f289 -m \"First commit\"" }
google/gemini-3-flash-preview:high PASS 100% 62 in → 196 out (182 reasoning)
git tag v0.1 1184181
google/gemini-3-flash-preview:high__json_schema PASS 100% 121 in → 335 out (318 reasoning)
git tag v0.1 d9edc13
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 d9edc13"}
google/gemini-3-flash-preview:low PASS 100% 63 in → 215 out (202 reasoning)
git tag v0.1 8328f65
google/gemini-3-flash-preview:low__json_schema PASS 100% 122 in → 265 out (247 reasoning)
git tag v0.1 0b9c404
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 0b9c404"}
google/gemini-3-flash-preview:medium PASS 100% 64 in → 251 out (237 reasoning)
git tag v0.1 7d2d784
google/gemini-3-flash-preview:medium__json_schema PASS 100% 118 in → 226 out (210 reasoning)
git tag v0.1 6febf50
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 6febf50"}
google/gemini-3-flash-preview:xhigh PASS 100% 64 in → 231 out (217 reasoning)
git tag v0.1 815c9f5
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 119 in → 193 out (175 reasoning)
git tag v0.1 1ec1418
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 1ec1418"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 62 in → 330 out (316 reasoning)
git tag v0.1 9157d3a
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 120 in → 677 out (651 reasoning)
git tag -a v0.1 b6d4d39 -m "First commit"
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 b6d4d39 -m \"First commit\""}
google/gemini-3.1-flash-lite-preview:low PASS 100% 66 in → 159 out (145 reasoning)
git tag v0.1 5820d21
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 63 in → 166 out (140 reasoning)
git tag -a v0.1 2cf2189 -m "First commit"
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 2cf2189 -m \"First commit\""}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 64 in → 223 out (210 reasoning)
git tag v0.1 f3a249f
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 64 in → 269 out (237 reasoning)
git tag -a v0.1 23a3e79 -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 23a3e79 -m \"First commit\"" }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 66 in → 249 out (235 reasoning)
git tag v0.1 10197d5
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 64 in → 582 out (555 reasoning)
git tag -a v0.1 36e1190 -m "First commit"
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 36e1190 -m \"First commit\""}
google/gemini-3.1-pro-preview:high PASS 100% 64 in → 270 out (257 reasoning)
git tag v0.1 9bc3c3d
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 119 in → 242 out (226 reasoning)
git tag v0.1 3e0caf0
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 3e0caf0"}
google/gemini-3.1-pro-preview:low PASS 100% 62 in → 351 out (338 reasoning)
git tag v0.1 5e9a6ff
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 120 in → 131 out (112 reasoning)
git tag v0.1 68076a4
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 68076a4"}
google/gemini-3.1-pro-preview:medium PASS 100% 61 in → 232 out (221 reasoning)
git tag v0.1 dea3e7b
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 120 in → 119 out (103 reasoning)
git tag v0.1 def45e9
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 def45e9"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 65 in → 329 out (316 reasoning)
git tag v0.1 c515576
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 122 in → 131 out (107 reasoning)
git tag v0.1 5c2e555
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 5c2e555" }
google/gemini-3.5-flash-lite:high PASS 100% 66 in → 249 out (235 reasoning)
git tag v0.1 896d06d
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 64 in → 220 out (202 reasoning)
git tag v0.1 8dd2844
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 8dd2844"}
google/gemini-3.5-flash-lite:low PASS 100% 61 in → 13 out (0 reasoning)
`git tag v0.1 1cc2acf`
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 121 in → 27 out (0 reasoning)
git tag -a v0.1 47099b3 -m 'First commit'
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 47099b3 -m 'First commit'"}
google/gemini-3.5-flash-lite:max PASS 100% 63 in → 316 out (302 reasoning)
git tag v0.1 1f1c503
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 63 in → 268 out (250 reasoning)
git tag v0.1 05bc30d
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 05bc30d"}
google/gemini-3.5-flash-lite:medium PASS 100% 62 in → 152 out (139 reasoning)
git tag v0.1 5bb022c
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 64 in → 134 out (115 reasoning)
git tag v0.1 8f82e75
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 8f82e75"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 62 in → 273 out (262 reasoning)
git tag v0.1 baa3ceb
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 120 in → 620 out (603 reasoning)
git tag v0.1 8bfca89
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 8bfca89"}
google/gemini-3.5-flash:high PASS 100% 64 in → 280 out (266 reasoning)
git tag v0.1 459887e
google/gemini-3.5-flash:high__json_schema PASS 100% 121 in → 282 out (265 reasoning)
git tag v0.1 ac1516f
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 ac1516f"}
google/gemini-3.5-flash:low PASS 100% 64 in → 246 out (233 reasoning)
git tag v0.1 695ee8e
google/gemini-3.5-flash:low__json_schema PASS 100% 122 in → 150 out (131 reasoning)
git tag v0.1 3e71665
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 3e71665"}
google/gemini-3.5-flash:medium PASS 100% 65 in → 277 out (263 reasoning)
git tag v0.1 3c2a327
google/gemini-3.5-flash:medium__json_schema PASS 100% 116 in → 323 out (305 reasoning)
git tag v0.1 65eb85f
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 65eb85f"}
google/gemini-3.5-flash:xhigh PASS 100% 65 in → 356 out (342 reasoning)
git tag v0.1 6d282b3
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 121 in → 420 out (402 reasoning)
git tag v0.1 17024bd
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 17024bd"}
google/gemini-3.6-flash:high PASS 100% 62 in → 318 out (306 reasoning)
git tag v0.1 25faae6
google/gemini-3.6-flash:high__json_schema PASS 100% 64 in → 370 out (352 reasoning)
git tag v0.1 0e0f83c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 0e0f83c"}
google/gemini-3.6-flash:low PASS 100% 66 in → 165 out (152 reasoning)
git tag v0.1 f571f4e
google/gemini-3.6-flash:low__json_schema PASS 100% 120 in → 24 out (0 reasoning)
git tag v0.1 7535933
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 7535933" }
google/gemini-3.6-flash:max PASS 100% 65 in → 308 out (294 reasoning)
git tag v0.1 3a45e06
google/gemini-3.6-flash:max__json_schema PASS 100% 117 in → 225 out (210 reasoning)
git tag v0.1 3dbadf6
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 3dbadf6"}
google/gemini-3.6-flash:medium PASS 100% 66 in → 261 out (247 reasoning)
git tag v0.1 62c6f08
google/gemini-3.6-flash:medium__json_schema PASS 100% 117 in → 209 out (193 reasoning)
git tag v0.1 6bfdc9c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 6bfdc9c"}
google/gemini-3.6-flash:xhigh PASS 100% 63 in → 259 out (247 reasoning)
git tag v0.1 1abb247
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 65 in → 314 out (297 reasoning)
git tag v0.1 447dc9c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 447dc9c"}
google/gemma-4-26b-a4b-it:high PASS 100% 81 in → 927 out (912 reasoning)
git tag v0.1 866190a
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 78 in → 823 out (651 reasoning)
git tag v0.1 0040cca
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 0040cca" }
google/gemma-4-26b-a4b-it:low PASS 100% 80 in → 1,564 out (1,305 reasoning)
git tag v0.1 a15b058
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 80 in → 244 out (149 reasoning)
git tag v0.1 7d37003
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 7d37003" }
google/gemma-4-26b-a4b-it:medium PASS 100% 80 in → 603 out (448 reasoning)
git tag v0.1 5b50478
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 80 in → 247 out (167 reasoning)
git tag v0.1 e74039e
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 e74039e" }
google/gemma-4-26b-a4b-it:none PASS 100% 75 in → 14 out (0 reasoning)
git tag v0.1 29984be
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 75 in → 24 out (0 reasoning)
git tag v0.1 65bfe4f
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 65bfe4f" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 80 in → 1,608 out (1,316 reasoning)
git tag v0.1 134ae07
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 77 in → 960 out (801 reasoning)
git tag v0.1 aaecefa
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 aaecefa"}
google/gemma-4-31b-it:high PASS 100% 79 in → 199 out (146 reasoning)
git tag v0.1 73079c9
google/gemma-4-31b-it:high__json_schema PASS 100% 77 in → 203 out (132 reasoning)
git tag v0.1 5be2ca8
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 5be2ca8" }
google/gemma-4-31b-it:low PASS 100% 83 in → 188 out (173 reasoning)
git tag v0.1 9f2f83b
google/gemma-4-31b-it:low__json_schema PASS 100% 79 in → 200 out (133 reasoning)
git tag v0.1 f734a8e
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 f734a8e" }
google/gemma-4-31b-it:medium PASS 100% 79 in → 211 out (153 reasoning)
git tag v0.1 c9622c9
google/gemma-4-31b-it:medium__json_schema PASS 100% 78 in → 182 out (123 reasoning)
git tag v0.1 26bcd44
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 26bcd44"}
google/gemma-4-31b-it:none PASS 100% 73 in → 13 out (0 reasoning)
git tag v0.1 75eee81
google/gemma-4-31b-it:none__json_schema PASS 100% 76 in → 20 out (0 reasoning)
git tag v0.1 8c20736
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 8c20736"}
google/gemma-4-31b-it:xhigh PASS 100% 80 in → 210 out (143 reasoning)
git tag v0.1 b4c6d22
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 80 in → 184 out (108 reasoning)
git tag v0.1 27601d0
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 27601d0" }
ibm-granite/granite-4.1-8b PASS 100% 67 in → 22 out
git tag -a v0.1 0b9d88a -m "First commit"
git tag -a v0.1 042d24b -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 042d24b -m 'First commit'" }
inclusionai/ling-2.6-flash PASS 100% 84 in → 18 out
``` git tag v0.1 d35b61f ```
git tag v0.1 9619fe3
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 9619fe3" }
minimax/minimax-m2.5:high PASS 100% 89 in → 228 out (203 reasoning)
git tag v0.1 069053b
minimax/minimax-m2.5:high__json_schema PASS 100% 99 in → 396 out (369 reasoning)
git tag -a v0.1 e1a88 -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 e1a88 -m \"First commit\"" }
minimax/minimax-m2.5:low PASS 100% 95 in → 85 out (64 reasoning)
git tag v0.1 2312d7d
minimax/minimax-m2.5:low__json_schema PASS 100% 91 in → 91 out (73 reasoning)
git tag v0.1 e5da060
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 e5da060" }
minimax/minimax-m2.5:medium PASS 100% 92 in → 322 out (311 reasoning)
git tag v0.1 749fb73
minimax/minimax-m2.5:medium__json_schema PASS 100% 97 in → 170 out (138 reasoning)
git tag -a v0.1 c0adb27 -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 c0adb27 -m 'First commit'" }
minimax/minimax-m2.5:xhigh PASS 100% 92 in → 137 out (102 reasoning)
git tag v0.1 19c498f
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 91 in → 125 out (99 reasoning)
git tag v0.1 c33f35d
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 c33f35d" }
minimax/minimax-m2.7:high PASS 100% 76 in → 134 out (94 reasoning)
git tag v0.1 63bfc24
minimax/minimax-m2.7:high__json_schema PASS 100% 90 in → 562 out (519 reasoning)
git tag v0.1 915d253
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 915d253" }
minimax/minimax-m2.7:low PASS 100% 93 in → 248 out (211 reasoning)
git tag v0.1 2ef7e6d
minimax/minimax-m2.7:medium PASS 100% 94 in → 643 out (632 reasoning)
git tag v0.1 70ca920
minimax/minimax-m2.7:medium__json_schema PASS 100% 222 in → 236 out (211 reasoning)
git tag -a v0.1 8733596 -m "First commit"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 8733596 -m \"First commit\""}
minimax/minimax-m2.7:xhigh PASS 100% 73 in → 707 out (689 reasoning)
git tag -a v0.1 bd40382 -m "First commit"
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 223 in → 302 out (277 reasoning)
git tag -a v0.1 78e8776 -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 78e8776 -m 'First commit'"}
minimax/minimax-m3:high PASS 100% 234 in → 67 out (50 reasoning)
git tag v0.1 d4c8dd5
minimax/minimax-m3:high__json_schema PASS 100% 216 in → 18 out (0 reasoning)
git tag v0.1 93839af
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 93839af" }
minimax/minimax-m3:low PASS 100% 229 in → 64 out (38 reasoning)
git tag v0.1 77f0cb6
minimax/minimax-m3:medium PASS 100% 231 in → 70 out (56 reasoning)
git tag v0.1 78be5d3
minimax/minimax-m3:medium__json_schema PASS 100% 214 in → 16 out (0 reasoning)
git tag v0.1 4608215
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 4608215"}
minimax/minimax-m3:xhigh PASS 100% 228 in → 82 out (67 reasoning)
git tag v0.1 f156e06
minimax/minimax-m3:xhigh__json_schema PASS 100% 177 in → 27 out (26 reasoning)
git tag -a v0.1 4edbd15 -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 4edbd15 -m 'First commit'" }
mistralai/devstral-2512 PASS 100% 64 in → 24 out
```bash git tag v0.1 30063de -m "First commit" ```
mistralai/devstral-2512__json_schema PASS 100% 62 in → 27 out
git tag -a v0.1 74bc098 -m 'First commit'
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 74bc098 -m 'First commit'"}
mistralai/mistral-medium-3-5:high PASS 100% 73 in → 239 out (210 reasoning)
git tag v0.1 cccd18c
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 63 in → 407 out (381 reasoning)
git tag v0.1 d775894
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 d775894"}
mistralai/mistral-medium-3-5:low PASS 100% 75 in → 325 out (281 reasoning)
git tag v0.1 b28bda2
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 63 in → 1,087 out (883 reasoning)
git tag v0.1 21f3e22
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 21f3e22"}
mistralai/mistral-medium-3-5:medium PASS 100% 75 in → 282 out (236 reasoning)
git tag v0.1 c699b84
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 65 in → 525 out (409 reasoning)
git tag v0.1 66d5e03
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 66d5e03"}
mistralai/mistral-medium-3-5:none PASS 100% 75 in → 20 out (0 reasoning)
git tag v0.1 f73f68b -m 'First commit'
mistralai/mistral-medium-3-5:xhigh PASS 100% 73 in → 620 out (541 reasoning)
git tag v0.1 19f5adb
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 65 in → 661 out (557 reasoning)
git tag v0.1 05f2a9f
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 05f2a9f"}
moonshotai/kimi-k2.6:high PASS 100% 62 in → 306 out (293 reasoning)
git tag v0.1 214bb2b
moonshotai/kimi-k2.6:high__json_schema PASS 100% 59 in → 490 out (323 reasoning)
git tag v0.1 4df812e
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 4df812e" }
moonshotai/kimi-k2.6:low PASS 100% 63 in → 671 out (659 reasoning)
git tag v0.1 08daf23
moonshotai/kimi-k2.6:medium PASS 100% 59 in → 435 out (395 reasoning)
git tag v0.1 78e1ddb
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 59 in → 275 out (256 reasoning)
git tag v0.1 902bc24
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 902bc24" }
moonshotai/kimi-k2.6:xhigh PASS 100% 62 in → 716 out (703 reasoning)
git tag v0.1 4e4b207
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 61 in → 366 out (329 reasoning)
git tag v0.1 cc1171a
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 cc1171a" }
moonshotai/kimi-k2.7-code:high PASS 100% 62 in → 183 out (170 reasoning)
git tag v0.1 a39af6e
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 61 in → 316 out (263 reasoning)
git tag v0.1 3797d3d
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 3797d3d"}
moonshotai/kimi-k2.7-code:low PASS 100% 63 in → 103 out (89 reasoning)
git tag v0.1 85d9ac1
moonshotai/kimi-k2.7-code:medium PASS 100% 63 in → 178 out (166 reasoning)
git tag v0.1 a75efb4
moonshotai/kimi-k2.7-code:xhigh PASS 100% 63 in → 83 out (69 reasoning)
git tag v0.1 db1f6e9
moonshotai/kimi-k3:high PASS 100% 141 in → 103 out (75 reasoning)
git tag v0.1 1f04c2d
moonshotai/kimi-k3:high__json_schema PASS 100% 139 in → 130 out (103 reasoning)
git tag v0.1 16a8a20
moonshotai/kimi-k3:low PASS 100% 142 in → 85 out (59 reasoning)
git tag v0.1 cebba6b
moonshotai/kimi-k3:low__json_schema PASS 100% 139 in → 97 out (71 reasoning)
git tag v0.1 89dbfd6
moonshotai/kimi-k3:max PASS 100% 138 in → 99 out (73 reasoning)
git tag v0.1 544c3de
moonshotai/kimi-k3:max__json_schema PASS 100% 139 in → 92 out (66 reasoning)
git tag v0.1 6a68eaf
moonshotai/kimi-k3:medium PASS 100% 140 in → 97 out (70 reasoning)
git tag v0.1 f2c19f6
moonshotai/kimi-k3:medium__json_schema PASS 100% 138 in → 90 out (64 reasoning)
git tag v0.1 85f6722
moonshotai/kimi-k3:xhigh PASS 100% 140 in → 97 out (71 reasoning)
git tag v0.1 a3942d2
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 237 in → 107 out (77 reasoning)
git tag v0.1 fca861e
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 fca861e"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 77 in → 187 out (136 reasoning)
git tag v0.1 4fb8045
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 76 in → 237 out (187 reasoning)
git tag v0.1 3d853e7
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 3d853e7" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 75 in → 229 out (171 reasoning)
git tag v0.1 43e5c2c
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 77 in → 568 out (484 reasoning)
git tag v0.1 705912f
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 705912f" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 78 in → 23 out (0 reasoning)
git tag -a v0.1 -m "First commit" 413f558
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 77 in → 36 out (0 reasoning)
git tag -a v0.1 12a3c75 -m 'First commit' && git push --tags
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 12a3c75 -m 'First commit' && git push --tags" }
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 76 in → 338 out (289 reasoning)
git tag v0.1 65aacb5
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 65aacb5" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 74 in → 300 out (250 reasoning)
git tag v0.1 aa16366
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 75 in → 141 out (110 reasoning)
git tag v0.1 a9995c3
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 a9995c3" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 77 in → 311 out (276 reasoning)
git tag v0.1 060b61b
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 76 in → 155 out (119 reasoning)
git tag -a v0.1 -m "First commit" ae93e16
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 -m \"First commit\" ae93e16" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 77 in → 338 out (320 reasoning)
git tag v0.1 4b02013
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 75 in → 183 out (140 reasoning)
git tag -a v0.1 ec52066 -m "First commit"
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 ec52066 -m \"First commit\"" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 77 in → 24 out (0 reasoning)
git tag -a v0.1 -m "v0.1" 30063de
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 74 in → 20 out (0 reasoning)
git tag v0.1 dfcbbf3
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 dfcbbf3" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 76 in → 124 out (84 reasoning)
git tag -a v0.1 02759ab -m "First commit"
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 76 in → 208 out (165 reasoning)
git tag -a v0.1 7a391d9 -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 7a391d9 -m 'First commit'" }
openai/gpt-5.4-mini:high PASS 100% 61 in → 270 out (246 reasoning)
git tag -a v0.1 de9a810 -m 'First commit'
openai/gpt-5.4-mini:high__json_schema PASS 100% 94 in → 108 out (83 reasoning)
git tag v0.1 1007aa1
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 1007aa1"}
openai/gpt-5.4-mini:low PASS 100% 59 in → 46 out (21 reasoning)
git tag -a v0.1 94da8ff -m "First commit"
openai/gpt-5.4-mini:low__json_schema PASS 100% 91 in → 105 out (71 reasoning)
git tag -a v0.1 86743ba -m "v0.1"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 86743ba -m \"v0.1\""}
openai/gpt-5.4-mini:medium PASS 100% 59 in → 62 out (45 reasoning)
git tag v0.1 5032ee8
openai/gpt-5.4-mini:medium__json_schema PASS 100% 93 in → 174 out (148 reasoning)
git tag v0.1 50e1cd3
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 50e1cd3"}
openai/gpt-5.4-mini:none PASS 100% 60 in → 22 out (0 reasoning)
git tag -a v0.1 f83af83 -m "First commit"
openai/gpt-5.4-mini:none__json_schema PASS 100% 98 in → 24 out (0 reasoning)
git tag v0.1 d73c6f3
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d73c6f3"}
openai/gpt-5.4-mini:xhigh PASS 100% 1,106 in → 3,631 out (3,631 reasoning)
git tag v0.1 ede2d8b
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 94 in → 540 out (516 reasoning)
git tag v0.1 6627906
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 6627906"}
openai/gpt-5.4-nano:high PASS 100% 59 in → 13 out (0 reasoning)
git tag v0.1 af60014
openai/gpt-5.4-nano:high__json_schema PASS 100% 95 in → 62 out (37 reasoning)
git tag v0.1 898fa8b
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 898fa8b"}
openai/gpt-5.4-nano:low PASS 100% 61 in → 14 out (0 reasoning)
git tag v0.1 aa5243b
openai/gpt-5.4-nano:low__json_schema PASS 100% 94 in → 23 out (0 reasoning)
git tag v0.1 5d04094
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 5d04094"}
openai/gpt-5.4-nano:medium PASS 100% 60 in → 15 out (0 reasoning)
git tag v0.1 791cce6
openai/gpt-5.4-nano:medium__json_schema PASS 100% 94 in → 32 out (0 reasoning)
git tag -a v0.1 d45ed20 -m "v0.1"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 d45ed20 -m \"v0.1\""}
openai/gpt-5.4-nano:none PASS 100% 59 in → 16 out (0 reasoning)
git tag v0.1 85d16a7
openai/gpt-5.4-nano:none__json_schema PASS 100% 96 in → 24 out (0 reasoning)
git tag v0.1 be8e7f1
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 be8e7f1"}
openai/gpt-5.4-nano:xhigh PASS 100% 59 in → 69 out (52 reasoning)
git tag v0.1 091d81c
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 96 in → 156 out (128 reasoning)
git tag v0.1 2f8a0f0
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 2f8a0f0"}
openai/gpt-5.4:high PASS 100% 61 in → 108 out (91 reasoning)
git tag v0.1 30238b8
openai/gpt-5.4:high__json_schema PASS 100% 93 in → 83 out (58 reasoning)
git tag v0.1 09454e3
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 09454e3"}
openai/gpt-5.4:low PASS 100% 61 in → 54 out (37 reasoning)
git tag v0.1 b2151c7
openai/gpt-5.4:low__json_schema PASS 100% 97 in → 62 out (36 reasoning)
git tag v0.1 db3e0f3
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 db3e0f3"}
openai/gpt-5.4:medium PASS 100% 59 in → 87 out (69 reasoning)
git tag v0.1 9f7cacd
openai/gpt-5.4:medium__json_schema PASS 100% 96 in → 62 out (38 reasoning)
git tag v0.1 7926459
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 7926459"}
openai/gpt-5.4:none PASS 100% 61 in → 26 out (0 reasoning)
git tag -a v0.1 1baf2ff -m "v0.1"
openai/gpt-5.4:none__json_schema PASS 100% 94 in → 22 out (0 reasoning)
git tag v0.1 041da87
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 041da87"}
openai/gpt-5.4:xhigh PASS 100% 58 in → 128 out (112 reasoning)
git tag v0.1 a943ac0
openai/gpt-5.4:xhigh__json_schema PASS 100% 93 in → 539 out (516 reasoning)
git tag v0.1 653beda
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 653beda"}
openai/gpt-5.5:high PASS 100% 63 in → 69 out (51 reasoning)
git tag v0.1 d3a4dcb
openai/gpt-5.5:high__json_schema PASS 100% 93 in → 156 out (132 reasoning)
git tag v0.1 1298714
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 1298714"}
openai/gpt-5.5:low PASS 100% 59 in → 49 out (32 reasoning)
git tag v0.1 3f96047
openai/gpt-5.5:low__json_schema PASS 100% 96 in → 25 out (0 reasoning)
git tag v0.1 4e5dd9a
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 4e5dd9a"}
openai/gpt-5.5:medium PASS 100% 61 in → 59 out (43 reasoning)
git tag v0.1 b454cc8
openai/gpt-5.5:medium__json_schema PASS 100% 96 in → 73 out (49 reasoning)
git tag v0.1 972bd53
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 972bd53"}
openai/gpt-5.5:none PASS 100% 58 in → 14 out (0 reasoning)
git tag v0.1 ecf8867
openai/gpt-5.5:none__json_schema PASS 100% 94 in → 24 out (0 reasoning)
git tag v0.1 74d0fc0
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 74d0fc0"}
openai/gpt-5.5:xhigh PASS 100% 58 in → 87 out (71 reasoning)
git tag v0.1 b38669c
openai/gpt-5.5:xhigh__json_schema PASS 100% 95 in → 96 out (72 reasoning)
git tag v0.1 649beaa
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 649beaa"}
openai/gpt-5.6-luna:high PASS 100% 60 in → 78 out (62 reasoning)
git tag v0.1 d78dde8
openai/gpt-5.6-luna:high__json_schema PASS 100% 92 in → 109 out (77 reasoning)
git tag -a v0.1 943185e -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 943185e -m 'First commit'"}
openai/gpt-5.6-luna:low PASS 100% 62 in → 77 out (53 reasoning)
git tag -a v0.1 a70cd91 -m "First commit"
openai/gpt-5.6-luna:low__json_schema PASS 100% 94 in → 111 out (79 reasoning)
git tag -a v0.1 e32f809 -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 e32f809 -m 'First commit'"}
openai/gpt-5.6-luna:max PASS 100% 61 in → 254 out (236 reasoning)
git tag v0.1 62d8b78
openai/gpt-5.6-luna:max__json_schema PASS 100% 93 in → 400 out (375 reasoning)
git tag v0.1 edd1b11
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 edd1b11"}
openai/gpt-5.6-luna:medium PASS 100% 61 in → 91 out (66 reasoning)
git tag -a v0.1 23fcdd2 -m "First commit"
openai/gpt-5.6-luna:medium__json_schema PASS 100% 94 in → 97 out (65 reasoning)
git tag -a v0.1 bf4181f -m "First commit"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 bf4181f -m \"First commit\""}
openai/gpt-5.6-luna:none PASS 100% 63 in → 24 out (0 reasoning)
git tag -a v0.1 f8d25e1 -m "First commit"
openai/gpt-5.6-luna:none__json_schema PASS 100% 94 in → 30 out (0 reasoning)
git tag -a v0.1 cf6836f -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 cf6836f -m 'First commit'"}
openai/gpt-5.6-luna:xhigh PASS 100% 61 in → 115 out (91 reasoning)
git tag -a v0.1 aa5e660 -m 'First commit'
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 95 in → 138 out (105 reasoning)
git tag -a v0.1 81280f7 -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 81280f7 -m 'First commit'"}
openai/gpt-5.6-sol:high PASS 100% 58 in → 61 out (45 reasoning)
git tag v0.1 ce7ee65
openai/gpt-5.6-sol:high__json_schema PASS 100% 96 in → 24 out (0 reasoning)
git tag v0.1 9b355e5
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 9b355e5"}
openai/gpt-5.6-sol:low PASS 100% 57 in → 13 out (0 reasoning)
git tag v0.1 c909873
openai/gpt-5.6-sol:low__json_schema PASS 100% 95 in → 22 out (0 reasoning)
git tag v0.1 dcc7381
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 dcc7381"}
openai/gpt-5.6-sol:max PASS 100% 57 in → 135 out (118 reasoning)
git tag v0.1 10710b7
openai/gpt-5.6-sol:max__json_schema PASS 100% 95 in → 196 out (170 reasoning)
git tag v0.1 3b18c45
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 3b18c45"}
openai/gpt-5.6-sol:medium PASS 100% 60 in → 23 out (0 reasoning)
git tag -a v0.1 aa4e4cd -m "First commit"
openai/gpt-5.6-sol:medium__json_schema PASS 100% 97 in → 32 out (0 reasoning)
git tag -a v0.1 f9e19b0 -m "First commit"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 f9e19b0 -m \"First commit\""}
openai/gpt-5.6-sol:none PASS 100% 64 in → 15 out (0 reasoning)
git tag v0.1 d42bcd9
openai/gpt-5.6-sol:none__json_schema PASS 100% 94 in → 30 out (0 reasoning)
git tag -a v0.1 6153625 -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 6153625 -m 'First commit'"}
openai/gpt-5.6-sol:xhigh PASS 100% 60 in → 14 out (0 reasoning)
git tag v0.1 e90608c
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 96 in → 25 out (0 reasoning)
git tag v0.1 7b0df0e
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 7b0df0e"}
openai/gpt-5.6-terra:high PASS 100% 59 in → 15 out (0 reasoning)
git tag v0.1 9cc883a
openai/gpt-5.6-terra:high__json_schema PASS 100% 92 in → 22 out (0 reasoning)
git tag v0.1 d89616e
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d89616e"}
openai/gpt-5.6-terra:low PASS 100% 62 in → 16 out (0 reasoning)
git tag v0.1 72fa4c4
openai/gpt-5.6-terra:low__json_schema PASS 100% 98 in → 24 out (0 reasoning)
git tag v0.1 d8a2dd7
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 d8a2dd7"}
openai/gpt-5.6-terra:max PASS 100% 62 in → 15 out (0 reasoning)
git tag v0.1 74d973e
openai/gpt-5.6-terra:max__json_schema PASS 100% 95 in → 68 out (44 reasoning)
git tag v0.1 4acf504
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 4acf504"}
openai/gpt-5.6-terra:medium PASS 100% 61 in → 14 out (0 reasoning)
git tag v0.1 0344357
openai/gpt-5.6-terra:medium__json_schema PASS 100% 95 in → 23 out (0 reasoning)
git tag v0.1 5fc858c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 5fc858c"}
openai/gpt-5.6-terra:none PASS 100% 61 in → 16 out (0 reasoning)
git tag v0.1 9fd48e1
openai/gpt-5.6-terra:none__json_schema PASS 100% 95 in → 23 out (0 reasoning)
git tag v0.1 e4cca8d
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 e4cca8d"}
openai/gpt-5.6-terra:xhigh PASS 100% 62 in → 16 out (0 reasoning)
git tag v0.1 da3d5f8
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 95 in → 22 out (0 reasoning)
git tag v0.1 fa13aef
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 fa13aef"}
openai/gpt-oss-120b:high PASS 100% 120 in → 564 out (503 reasoning)
git tag -a v0.1 515d0b0 -m "First commit"
openai/gpt-oss-120b:high__json_schema PASS 100% 108 in → 285 out (222 reasoning)
git tag -a v0.1 b5b92fa -m "First commit"
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 b5b92fa -m \"First commit\""}
openai/gpt-oss-120b:low PASS 100% 122 in → 73 out (31 reasoning)
git tag -a v0.1 c8501f0 -m "First commit"
openai/gpt-oss-120b:low__json_schema PASS 100% 106 in → 82 out (37 reasoning)
git tag -a v0.1 71252f0 -m "First commit"
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 71252f0 -m \"First commit\""}
openai/gpt-oss-120b:medium PASS 100% 106 in → 249 out (197 reasoning)
git tag -a v0.1 -m "First commit" 2ab26ea
openai/gpt-oss-120b:medium__json_schema PASS 100% 120 in → 141 out (101 reasoning)
git tag v0.1 ec87ab4
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 ec87ab4"}
openai/gpt-oss-120b:xhigh PASS 100% 121 in → 367 out (325 reasoning)
git tag v0.1 d17fc3e
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 109 in → 875 out (784 reasoning)
git tag v0.1 7c9a20c
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 7c9a20c" }
openai/gpt-oss-20b:high PASS 100% 107 in → 305 out (277 reasoning)
git tag v0.1 7855c39
openai/gpt-oss-20b:high__json_schema PASS 100% 109 in → 321 out (241 reasoning)
git tag v0.1 3b7904b
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 3b7904b"}
openai/gpt-oss-20b:low PASS 100% 110 in → 147 out (82 reasoning)
git tag -a v0.1 50b0f06 -m "First commit"
openai/gpt-oss-20b:low__json_schema PASS 100% 173 in → 107 out (71 reasoning)
git tag -a v0.1 -m "First commit" 4e61268
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 -m \"First commit\" 4e61268"}
openai/gpt-oss-20b:medium PASS 100% 122 in → 293 out (223 reasoning)
git tag -a v0.1 c19d652 -m "First commit"
openai/gpt-oss-20b:medium__json_schema PASS 100% 107 in → 377 out (338 reasoning)
git tag v0.1 36c5934
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 36c5934"}
openai/gpt-oss-20b:xhigh PASS 100% 119 in → 902 out (823 reasoning)
git tag v0.1 46fbdc7
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 122 in → 180 out (145 reasoning)
git tag v0.1 590a45c
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 590a45c" }
poolside/laguna-m.1:high PASS 100% 71 in → 685 out (675 reasoning)
git tag v0.1 dfcfada
poolside/laguna-m.1:high__json_schema PASS 100% 76 in → 441 out (415 reasoning)
git tag -a v0.1 -m "First commit" 96a2f6b
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 -m \"First commit\" 96a2f6b"}
poolside/laguna-m.1:low PASS 100% 77 in → 476 out (462 reasoning)
git tag v0.1 0cd4540
poolside/laguna-m.1:low__json_schema PASS 100% 77 in → 1,019 out (1,003 reasoning)
git tag v0.1 bf63b97
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 bf63b97"}
poolside/laguna-m.1:medium PASS 100% 77 in → 1,122 out (1,112 reasoning)
git tag v0.1 9943668
poolside/laguna-m.1:medium__json_schema PASS 100% 77 in → 331 out (311 reasoning)
git tag v0.1 f55e725
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 f55e725"}
poolside/laguna-m.1:xhigh PASS 100% 78 in → 1,247 out (1,233 reasoning)
git tag v0.1 e60b8d4
poolside/laguna-m.1:xhigh__json_schema PASS 100% 77 in → 822 out (791 reasoning)
git tag -a v0.1 -m "First commit" 0f7908e
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 -m \"First commit\" 0f7908e" }
poolside/laguna-xs-2.1:high PASS 100% 76 in → 203 out (192 reasoning)
git tag v0.1 da4367b
poolside/laguna-xs-2.1:high__json_schema PASS 100% 78 in → 574 out (550 reasoning)
git tag v0.1 7d0f5c0
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 7d0f5c0" }
poolside/laguna-xs-2.1:low PASS 100% 78 in → 344 out (331 reasoning)
git tag v0.1 42b0108
poolside/laguna-xs-2.1:low__json_schema PASS 100% 79 in → 178 out (152 reasoning)
git tag v0.1 618398c
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 618398c" }
poolside/laguna-xs-2.1:medium PASS 100% 75 in → 191 out (179 reasoning)
git tag v0.1 d3382b4
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 78 in → 232 out (214 reasoning)
git tag v0.1 12bf0d0
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 12bf0d0"}
poolside/laguna-xs-2.1:none PASS 100% 76 in → 20 out (0 reasoning)
git tag -a v0.1 -m 'First commit' ff8774f
poolside/laguna-xs-2.1:none__json_schema PASS 100% 78 in → 28 out (0 reasoning)
git tag -a v0.1 -m 'First commit' 5b1c320
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 -m 'First commit' 5b1c320"}
poolside/laguna-xs-2.1:xhigh PASS 100% 78 in → 281 out (271 reasoning)
git tag v0.1 ff63592
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 77 in → 231 out (207 reasoning)
git tag v0.1 981d562
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 981d562" }
poolside/laguna-xs.2:high__json_schema PASS 100% 113 in → 122 out (101 reasoning)
git tag -f v0.1 ef30741
JSON Schema Structured Output
(raw) {"command": "git tag -f v0.1 ef30741"}
poolside/laguna-xs.2:low PASS 100% 113 in → 127 out (112 reasoning)
git tag v0.1 c7b8dd8
poolside/laguna-xs.2:low__json_schema PASS 100% 115 in → 80 out (59 reasoning)
git tag v0.1 0590b13
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 0590b13"}
poolside/laguna-xs.2:medium PASS 100% 113 in → 350 out (334 reasoning)
git tag v0.1 f9c8087
poolside/laguna-xs.2:medium__json_schema PASS 100% 112 in → 606 out (579 reasoning)
git tag -a v0.1 8dbc23c -m "First commit"
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 8dbc23c -m \"First commit\""}
poolside/laguna-xs.2:none PASS 100% 115 in → 16 out (0 reasoning)
git tag v0.1 48972d6
poolside/laguna-xs.2:none__json_schema PASS 100% 114 in → 20 out (0 reasoning)
git tag v0.1 773558a
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 773558a"}
poolside/laguna-xs.2:xhigh PASS 100% 112 in → 199 out (186 reasoning)
git tag v0.1 ee0866c
qwen/qwen3.6-27b:high PASS 100% 74 in → 1,172 out (947 reasoning)
git tag v0.1 f977a56
qwen/qwen3.6-27b:high__json_schema PASS 100% 73 in → 1,017 out (1 reasoning)
git tag v0.1 ea5f6b5
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 ea5f6b5" }
qwen/qwen3.6-27b:low PASS 100% 74 in → 1,004 out (1 reasoning)
git tag v0.1 fc4a802
qwen/qwen3.6-27b:low__json_schema PASS 100% 72 in → 1,098 out (934 reasoning)
git tag v0.1 b956ba1
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 b956ba1" }
qwen/qwen3.6-27b:medium PASS 100% 72 in → 894 out (723 reasoning)
git tag v0.1 8dc6057
qwen/qwen3.6-27b:medium__json_schema PASS 100% 75 in → 1,241 out (1,015 reasoning)
git tag v0.1 f1a6122
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 f1a6122" }
qwen/qwen3.6-27b:none PASS 100% 74 in → 14 out (0 reasoning)
git tag v0.1 4ba9a2e
qwen/qwen3.6-27b:xhigh PASS 100% 74 in → 767 out (589 reasoning)
git tag v0.1 74f4154
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 72 in → 959 out (764 reasoning)
git tag v0.1 6031ee9
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 6031ee9" }
qwen/qwen3.6-35b-a3b:high PASS 100% 75 in → 712 out (557 reasoning)
git tag v0.1 8d50517
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 73 in → 987 out (757 reasoning)
git tag v0.1 1d92033
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 1d92033" }
qwen/qwen3.6-35b-a3b:low PASS 100% 73 in → 372 out (298 reasoning)
git tag v0.1 9be2b6b
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 74 in → 878 out (651 reasoning)
git tag v0.1 5489909
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 5489909" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 70 in → 1,112 out (1,093 reasoning)
git tag v0.1 515088c
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 72 in → 1,090 out (876 reasoning)
git tag v0.1 61ca582
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 61ca582" }
qwen/qwen3.6-35b-a3b:none PASS 100% 75 in → 14 out (0 reasoning)
git tag v0.1 657dca4
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 72 in → 813 out (796 reasoning)
git tag v0.1 6d87fea
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 73 in → 802 out (622 reasoning)
git tag v0.1 1122ed5
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 1122ed5" }
qwen/qwen3.6-flash:high PASS 100% 72 in → 958 out (941 reasoning)
git tag v0.1 3acbb3a
qwen/qwen3.6-flash:low PASS 100% 73 in → 1,004 out (985 reasoning)
git tag v0.1 711a59e
qwen/qwen3.6-flash:medium PASS 100% 72 in → 340 out (321 reasoning)
git tag v0.1 9e274e9
qwen/qwen3.6-flash:none PASS 100% 76 in → 14 out (0 reasoning)
git tag v0.1 91a8a9d
qwen/qwen3.6-flash:xhigh PASS 100% 74 in → 254 out (237 reasoning)
git tag v0.1 bd91f1b
qwen/qwen3.7-flash:high PASS 100% 75 in → 1,105 out (1,086 reasoning)
git tag v0.1 684a781
qwen/qwen3.7-flash:low PASS 100% 71 in → 401 out (384 reasoning)
git tag v0.1 39e8cea
qwen/qwen3.7-flash:max PASS 100% 72 in → 1,099 out (1,083 reasoning)
git tag v0.1 ce0bcf9
qwen/qwen3.7-flash:medium PASS 100% 72 in → 348 out (332 reasoning)
git tag v0.1 cfc4dd0
qwen/qwen3.7-flash:none PASS 100% 76 in → 12 out (0 reasoning)
git tag v0.1 d1cfd6c
qwen/qwen3.7-flash:xhigh PASS 100% 75 in → 396 out (378 reasoning)
git tag v0.1 b277111
qwen/qwen3.7-max:high PASS 100% 73 in → 421 out (402 reasoning)
git tag v0.1 7f74940
qwen/qwen3.7-max:low PASS 100% 71 in → 229 out (210 reasoning)
git tag v0.1 615550b
qwen/qwen3.7-max:medium PASS 100% 74 in → 310 out (291 reasoning)
git tag v0.1 6f8860d
qwen/qwen3.7-max:none PASS 100% 77 in → 14 out (0 reasoning)
git tag v0.1 7d14579
qwen/qwen3.7-max:xhigh PASS 100% 72 in → 462 out (445 reasoning)
git tag v0.1 ecd027f
qwen/qwen3.7-plus:high PASS 100% 71 in → 547 out (531 reasoning)
git tag v0.1 f81fbcc
qwen/qwen3.7-plus:low PASS 100% 74 in → 631 out (612 reasoning)
git tag v0.1 866f628
qwen/qwen3.7-plus:medium PASS 100% 74 in → 427 out (408 reasoning)
git tag v0.1 1e9b356
qwen/qwen3.7-plus:none PASS 100% 75 in → 19 out (0 reasoning)
git tag v0.1 415fd21 -m 'First commit'
qwen/qwen3.7-plus:xhigh PASS 100% 71 in → 547 out (531 reasoning)
git tag v0.1 f58afc9
tencent/hy3:high PASS 100% 68 in → 1,613 out (1,592 reasoning)
git tag -a v0.1 c29afc2 -m "First commit"
tencent/hy3:high__json_schema PASS 100% 65 in → 1,910 out (1,687 reasoning)
git tag -a v0.1 bc4df7c -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 bc4df7c -m 'First commit'" }
tencent/hy3:low PASS 100% 66 in → 615 out (602 reasoning)
git tag v0.1 af0cfd0
tencent/hy3:low__json_schema PASS 100% 64 in → 443 out (385 reasoning)
git tag v0.1 b267906
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 b267906" }
tencent/hy3:medium PASS 100% 69 in → 303 out (287 reasoning)
git tag v0.1 9e3f3c3
tencent/hy3:medium__json_schema PASS 100% 64 in → 319 out (253 reasoning)
git tag v0.1 c4e5664
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 c4e5664" }
tencent/hy3:none PASS 100% 67 in → 19 out (0 reasoning)
git tag -m 'First commit' v0.1 7ed5fb4
tencent/hy3:none__json_schema PASS 100% 68 in → 26 out (0 reasoning)
git tag -a v0.1 d032d0c -m 'First commit'
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 d032d0c -m 'First commit'" }
tencent/hy3:xhigh PASS 100% 65 in → 717 out (697 reasoning)
git tag -a v0.1 -m "First commit" 284d765
tencent/hy3:xhigh__json_schema PASS 100% 64 in → 678 out (585 reasoning)
git tag v0.1 146316c
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 146316c" }
thinkingmachines/inkling-small:high PASS 100% 71 in → 163 out (145 reasoning)
git tag v0.1 57a6ef0
thinkingmachines/inkling-small:high__json_schema PASS 100% 68 in → 139 out (116 reasoning)
git tag v0.1 8abebce
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 8abebce" }
thinkingmachines/inkling-small:low PASS 100% 71 in → 55 out (37 reasoning)
git tag v0.1 3a1b077
thinkingmachines/inkling-small:low__json_schema PASS 100% 70 in → 62 out (39 reasoning)
git tag v0.1 0180d62
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 0180d62" }
thinkingmachines/inkling-small:max PASS 100% 73 in → 406 out (318 reasoning)
git tag v0.1 2dfcfb5
thinkingmachines/inkling-small:max__json_schema PASS 100% 69 in → 116 out (96 reasoning)
git tag v0.1 0110204
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 0110204"}
thinkingmachines/inkling-small:medium PASS 100% 70 in → 97 out (80 reasoning)
git tag v0.1 e9668f9
thinkingmachines/inkling-small:medium__json_schema PASS 100% 68 in → 99 out (79 reasoning)
git tag v0.1 162238f
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 162238f"}
thinkingmachines/inkling-small:xhigh PASS 100% 71 in → 199 out (180 reasoning)
git tag v0.1 a4e0e1c
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 70 in → 105 out (83 reasoning)
git tag v0.1 0df1cd6
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 0df1cd6"}
thinkingmachines/inkling:high PASS 100% 68 in → 120 out (103 reasoning)
git tag v0.1 2afaf05
thinkingmachines/inkling:high__json_schema PASS 100% 67 in → 77 out (53 reasoning)
git tag v0.1 01a2731
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 01a2731" }
thinkingmachines/inkling:low PASS 100% 69 in → 43 out (27 reasoning)
git tag v0.1 e79ae83
thinkingmachines/inkling:low__json_schema PASS 100% 68 in → 50 out (28 reasoning)
git tag v0.1 f347e23
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 f347e23" }
thinkingmachines/inkling:max PASS 100% 68 in → 104 out (76 reasoning)
git tag v0.1 6182d70
thinkingmachines/inkling:max__json_schema PASS 100% 67 in → 114 out (92 reasoning)
git tag v0.1 af6016d
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 af6016d" }
thinkingmachines/inkling:medium PASS 100% 70 in → 100 out (82 reasoning)
git tag v0.1 3d577be
thinkingmachines/inkling:medium__json_schema PASS 100% 68 in → 168 out (146 reasoning)
git tag v0.1 4410131
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 4410131" }
thinkingmachines/inkling:xhigh PASS 100% 67 in → 288 out (219 reasoning)
git tag v0.1 8b30ce7
thinkingmachines/inkling:xhigh__json_schema PASS 100% 68 in → 106 out (86 reasoning)
git tag v0.1 c26e492
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 c26e492"}
x-ai/grok-4.3:high PASS 100% 243 in → 651 out (638 reasoning)
git tag v0.1 3f5f26b
x-ai/grok-4.3:high__json_schema PASS 100% 294 in → 381 out (366 reasoning)
git tag v0.1 769be2c
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 769be2c"}
x-ai/grok-4.3:low PASS 100% 243 in → 135 out (124 reasoning)
git tag v0.1 96b43ff
x-ai/grok-4.3:low__json_schema PASS 100% 302 in → 402 out (386 reasoning)
git tag v0.1 2e2074b
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 2e2074b"}
x-ai/grok-4.3:max PASS 100% 239 in → 189 out (179 reasoning)
git tag v0.1 c74f253
x-ai/grok-4.3:max__json_schema PASS 100% 294 in → 823 out (808 reasoning)
git tag v0.1 c74e8da
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 c74e8da"}
x-ai/grok-4.3:medium PASS 100% 242 in → 220 out (210 reasoning)
git tag v0.1 e380ee9
x-ai/grok-4.3:medium__json_schema PASS 100% 299 in → 493 out (474 reasoning)
git tag v0.1 3b04685
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 3b04685" }
x-ai/grok-4.3:none PASS 100% 237 in → 18 out (0 reasoning)
git tag -a v0.1 423c857 -m 'First commit'
x-ai/grok-4.3:none__json_schema PASS 100% 293 in → 25 out (0 reasoning)
git tag -a v0.1 1e33c7a -m 'First commit'
JSON Schema Structured Output
(raw) {"command":"git tag -a v0.1 1e33c7a -m 'First commit'"}
x-ai/grok-4.3:xhigh PASS 100% 238 in → 798 out (787 reasoning)
git tag v0.1 1bdbb19
x-ai/grok-4.3:xhigh__json_schema PASS 100% 296 in → 846 out (832 reasoning)
git tag v0.1 115d734
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 115d734"}
x-ai/grok-4.5:high PASS 100% 261 in → 304 out (292 reasoning)
git tag v0.1 81a6ff1
x-ai/grok-4.5:high__json_schema PASS 100% 327 in → 648 out (632 reasoning)
git tag v0.1 a8c402b
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 a8c402b"}
x-ai/grok-4.5:low PASS 100% 257 in → 169 out (154 reasoning)
``` git tag v0.1 58557a9 ```
x-ai/grok-4.5:low__json_schema PASS 100% 324 in → 263 out (247 reasoning)
git tag v0.1 06f2154
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 06f2154"}
x-ai/grok-4.5:max PASS 100% 260 in → 253 out (241 reasoning)
git tag v0.1 3cb3cc6
x-ai/grok-4.5:max__json_schema PASS 100% 327 in → 309 out (292 reasoning)
git tag v0.1 2e149c1
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 2e149c1"}
x-ai/grok-4.5:medium PASS 100% 260 in → 200 out (189 reasoning)
git tag v0.1 89afd48
x-ai/grok-4.5:medium__json_schema PASS 100% 326 in → 461 out (444 reasoning)
git tag v0.1 8c139d9
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 8c139d9"}
x-ai/grok-4.5:xhigh PASS 100% 259 in → 228 out (218 reasoning)
git tag v0.1 5179100
x-ai/grok-4.5:xhigh__json_schema PASS 100% 327 in → 354 out (336 reasoning)
git tag v0.1 68e2d6e
JSON Schema Structured Output
(raw) {"command":"git tag v0.1 68e2d6e"}
z-ai/glm-4.7-flash:high PASS 100% 63 in → 852 out (752 reasoning)
git tag v0.1 1f7e459
z-ai/glm-4.7-flash:high__json_schema PASS 100% 59 in → 352 out (283 reasoning)
git tag -a v0.1 -m 'First commit' dadffc1
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 -m 'First commit' dadffc1"}
z-ai/glm-4.7-flash:low PASS 100% 62 in → 462 out (381 reasoning)
git tag -a v0.1 -m "First commit" 62a42d4
z-ai/glm-4.7-flash:low__json_schema PASS 100% 60 in → 362 out (287 reasoning)
git tag -a v0.1 -m "First commit" e21dc90
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 -m \"First commit\" e21dc90"}
z-ai/glm-4.7-flash:medium PASS 100% 61 in → 319 out (266 reasoning)
git tag v0.1 37cbfee
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 60 in → 413 out (333 reasoning)
git tag v0.1 2f0ed63
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 2f0ed63"}
z-ai/glm-4.7-flash:xhigh PASS 100% 59 in → 334 out (273 reasoning)
git tag v0.1 3872e37
z-ai/glm-5.2:high PASS 100% 63 in → 137 out (95 reasoning)
git tag v0.1 50523c7
z-ai/glm-5.2:high__json_schema PASS 100% 67 in → 311 out (262 reasoning)
git tag v0.1 91dfffb
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 91dfffb" }
z-ai/glm-5.2:low PASS 100% 68 in → 296 out (281 reasoning)
git tag v0.1 9f4c4df
z-ai/glm-5.2:low__json_schema PASS 100% 70 in → 89 out (56 reasoning)
git tag v0.1 9c7c521
JSON Schema Structured Output
(raw) { "command": "git tag v0.1 9c7c521" }
z-ai/glm-5.2:medium PASS 100% 67 in → 323 out (310 reasoning)
git tag v0.1 a6babe0
z-ai/glm-5.2:none PASS 100% 61 in → 11 out (0 reasoning)
git tag v0.1 d969ecc
z-ai/glm-5.2:xhigh PASS 100% 68 in → 303 out (246 reasoning)
git tag v0.1 f44a47a
z-ai/glm-5.2:xhigh__json_schema PASS 100% 70 in → 294 out (275 reasoning)
git tag v0.1 e4341e6
JSON Schema Structured Output
(raw) {"command": "git tag v0.1 e4341e6"}
poolside/laguna-m.1:none FAIL 50% 78 in → 23 out (0 reasoning)
git tag -a v0.1 -m 'First commit' d06e5f4
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
z-ai/glm-4.7-flash:none FAIL 50% 58 in → 19 out (0 reasoning)
git tag -a v0.1 817daf0 -m "First commit"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
z-ai/glm-4.7-flash:none__json_schema FAIL 50% 57 in → 24 out (0 reasoning)
git tag -a v0.1 a48ec50 -m 'First commit'
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 a48ec50 -m 'First commit'"}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh FAIL 0% 65 in → 1,265 out (1,252 reasoning)
git tag -a v0.1 8e3e7c5
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low__json_schema FAIL 0% 61 in → 517 out (515 reasoning)
git tag-m "First commit" v0.1 ad188ca
JSON Schema Structured Output
(raw) { "command": "git tag-m \"First commit\" v0.1 ad188ca" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.
liquid/lfm-2-24b-a2b FAIL 0% 62 in → 12 out
git tag v0.1 < 22841dc
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
Invalid JSON. Output: git tag v0.1 v0.1
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 78 in → 273 out (213 reasoning)
git tag -a v0.1 9463174
JSON Schema Structured Output
(raw) { "command": "git tag -a v0.1 9463174" }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
nvidia/nemotron-3-nano-30b-a3b:low FAIL 0% 75 in → 815 out (687 reasoning)
git tag -a v0.1 $(git log --oneline --grep='First commit' | cut -d' ' -f1)
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 75 in → 568 out (522 reasoning)
git tag -a v0.1 c8ce37c
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-xs.2:high FAIL 0% 116 in → 249 out (232 reasoning)
git tag -a v0.1 8f975f4
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 114 in → 136 out (114 reasoning)
git tag -a v0.1 79fa437
JSON Schema Structured Output
(raw) {"command": "git tag -a v0.1 79fa437"}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-076701a2-e858-9603-ad25-cd800caab7ef","request_id":"076701a2-e858-9603-ad25-cd800caab7ef"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-10fec43d-7666-915f-b22e-a4aab78c8049","request_id":"10fec43d-7666-915f-b22e-a4aab78c8049"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9e4f3167-896e-9414-acf2-aadd6a198524","request_id":"9e4f3167-896e-9414-acf2-aadd6a198524"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d0f17eb7-19cd-9df0-b562-da31806d3251","request_id":"d0f17eb7-19cd-9df0-b562-da31806d3251"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b9000eec-8e5d-9c1c-b6a4-8367b56c3958","request_id":"b9000eec-8e5d-9c1c-b6a4-8367b56c3958"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-68367c81-5b56-9f2e-bb90-fea819840ea7"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9be05070-2b4f-9cb3-b02d-13beb33e946d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-1900b9a9-fc10-95ea-8ff7-7d63ddcc2be1"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c0e5eea5-57b3-9c56-8940-5ba0b2a20ac8"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c04a818a-beaf-9ace-975f-ac04fdbbd5d0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b49c7d33-3a8b-9a01-a525-8249c5e00fdb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3f63d266-2c5f-93c3-ac5f-322c0ef13066","request_id":"3f63d266-2c5f-93c3-ac5f-322c0ef13066"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a74e1fa7-0970-9271-aa1f-a3833130f27c","request_id":"a74e1fa7-0970-9271-aa1f-a3833130f27c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9d935a1a-ee67-9cd9-a3d0-1b9c924bd10f","request_id":"9d935a1a-ee67-9cd9-a3d0-1b9c924bd10f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a848f8f2-32fc-942d-af52-b7130f75e333","request_id":"a848f8f2-32fc-942d-af52-b7130f75e333"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8ea5232d-5728-9ab1-a827-eb3b50805360","request_id":"8ea5232d-5728-9ab1-a827-eb3b50805360"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbbc2124-49d1-942a-bdcf-758ad246fa17","request_id":"fbbc2124-49d1-942a-bdcf-758ad246fa17"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-da73b90d-fbe2-9fbf-9d84-05fcf3631e2a","request_id":"da73b90d-fbe2-9fbf-9d84-05fcf3631e2a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bfe073fb-eca1-9f31-8547-10ffa3b8554e","request_id":"bfe073fb-eca1-9f31-8547-10ffa3b8554e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 70 in → 16 out (0 reasoning)
git tag v0.1 2d8d41a<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
Invalid JSON. Output: git tag v0.1 c253019{"command": "git tag v0.1 c253019"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 70 in → 15 out (0 reasoning)
git tag v0.1 545b0e2<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git tag -l', 'contains': 'v0.1'}, {'type': 'git_output', 'command': 'git log v0.1 --oneline -1', 'contains': 'First commit'}]
Invalid JSON. Output: git tag v0.1 abd57de{ "command": "git tag v0.1 abd57de" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git tag v0.1 4fdffed
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.