Remove untracked files in a specific directory
Tests ability to remove untracked files in a specific directory path. Evaluates scoped cleanup.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 mkdir -p src test
  5. 05 echo 'main' > src/main.py
  6. 06 git add src test
  7. 07 git commit -m 'Initial commit'
  8. 08 echo 'temp' > src/temp.py
  9. 09 echo 'fixture' > test/temp.py
  10. 10 echo 'root' > root.txt
Prompt
Remove untracked files only inside the 'src' directory, leaving other untracked files alone. Output ONLY the git command, nothing else.
Expected
git clean -f src/

Scoped model quality, cost, API time, and token usage for git_clean/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 112 in → 18 out (16 reasoning)
git clean -f src/
anthropic/claude-fable-5:high__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
anthropic/claude-fable-5:low PASS 100% 112 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-fable-5:low__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
anthropic/claude-fable-5:max PASS 100% 112 in → 261 out (47 reasoning)
``` git clean -f src/ ```
anthropic/claude-fable-5:max__json_schema PASS 100% 322 in → 446 out (73 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-fable-5:medium PASS 100% 112 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
anthropic/claude-fable-5:xhigh PASS 100% 112 in → 39 out (20 reasoning)
git clean -f src/
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
anthropic/claude-haiku-4.5:high PASS 100% 122 in → 313 out (273 reasoning)
```git git clean -f src/ ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 285 in → 661 out (521 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
anthropic/claude-haiku-4.5:low PASS 100% 122 in → 271 out (241 reasoning)
``` git clean -f src/ ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 285 in → 284 out (246 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-haiku-4.5:medium PASS 100% 122 in → 541 out (501 reasoning)
``` git clean -f src/ ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 285 in → 383 out (341 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
anthropic/claude-haiku-4.5:none PASS 100% 93 in → 14 out (0 reasoning)
```bash git clean -fd src/ ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 255 in → 14 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 122 in → 238 out (205 reasoning)
``` git clean -f src ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 285 in → 1,214 out (659 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
anthropic/claude-opus-4.6:high PASS 100% 93 in → 14 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 256 in → 59 out (31 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.6:low PASS 100% 93 in → 9 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.6:low__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.6:max PASS 100% 93 in → 14 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 256 in → 96 out (73 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.6:medium PASS 100% 93 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.6:none PASS 100% 93 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.6:xhigh PASS 100% 93 in → 14 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 256 in → 59 out (31 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.7:high PASS 100% 117 in → 11 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.7:high__json_schema PASS 100% 327 in → 18 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.7:low PASS 100% 117 in → 11 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.7:low__json_schema PASS 100% 327 in → 18 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.7:max PASS 100% 117 in → 13 out (0 reasoning)
`git clean -f src/`
anthropic/claude-opus-4.7:max__json_schema PASS 100% 327 in → 18 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.7:medium PASS 100% 117 in → 11 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 327 in → 17 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command":"git clean -f src"}
anthropic/claude-opus-4.7:none PASS 100% 117 in → 13 out (0 reasoning)
`git clean -f src/`
anthropic/claude-opus-4.7:none__json_schema PASS 100% 327 in → 18 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.7:xhigh PASS 100% 117 in → 13 out (0 reasoning)
`git clean -f src/`
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 327 in → 18 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.8:high PASS 100% 112 in → 9 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.8:high__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.8:low PASS 100% 112 in → 9 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.8:low__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.8:max PASS 100% 112 in → 438 out (48 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 322 in → 153 out (33 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-opus-4.8:medium PASS 100% 112 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.8:none PASS 100% 112 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-opus-4.8:xhigh PASS 100% 112 in → 9 out (0 reasoning)
git clean -f src/
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:high PASS 100% 93 in → 93 out (70 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:low PASS 100% 93 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:max PASS 100% 93 in → 126 out (97 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 256 in → 136 out (112 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:medium PASS 100% 93 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:none PASS 100% 93 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 93 in → 93 out (69 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 256 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-5:high PASS 100% 112 in → 13 out (0 reasoning)
``` git clean -f src/ ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-5:low PASS 100% 112 in → 9 out (0 reasoning)
git clean -f src/
anthropic/claude-sonnet-5:low__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-sonnet-5:medium PASS 100% 112 in → 8 out (0 reasoning)
git clean -f src
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 322 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
anthropic/claude-sonnet-5:none PASS 100% 112 in → 14 out (0 reasoning)
``` git clean -fd src/ ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
anthropic/claude-sonnet-5:xhigh PASS 100% 112 in → 14 out (0 reasoning)
``` git clean -fd src/ ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 322 in → 15 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
arcee-ai/trinity-large-thinking:high PASS 100% 73 in → 1,796 out (1,802 reasoning)
git clean -f src/
arcee-ai/trinity-large-thinking:low PASS 100% 73 in → 2,179 out (2,203 reasoning)
git clean -f src/
arcee-ai/trinity-large-thinking:medium PASS 100% 73 in → 4,563 out (4,577 reasoning)
git clean -f src/
arcee-ai/trinity-large-thinking:xhigh PASS 100% 73 in → 1,491 out (1,492 reasoning)
git clean -f -d src/
arcee-ai/trinity-mini:high__json_schema PASS 100% 73 in → 421 out (457 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
arcee-ai/trinity-mini:low PASS 100% 73 in → 292 out (306 reasoning)
git clean -f src
arcee-ai/trinity-mini:low__json_schema PASS 100% 73 in → 413 out (436 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 73 in → 438 out (480 reasoning)
git clean -f -d src
JSON Schema Structured Output
(raw) { "command": "git clean -f -d src" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 74 in → 204 out (207 reasoning)
git clean -fd src
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 153 in → 1,959 out (1,958 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 74 in → 391 out (384 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 74 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 153 in → 149 out (144 reasoning)
git clean -fd src
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 153 in → 180 out (168 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
deepseek/deepseek-v4-flash-0731:medium PASS 100% 74 in → 1,317 out (1,364 reasoning)
git clean -f src
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 74 in → 429 out (412 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src/" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 74 in → 7 out (0 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 74 in → 14 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 153 in → 2,377 out (2,370 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 74 in → 563 out (565 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src/" }
deepseek/deepseek-v4-flash:high PASS 100% 74 in → 506 out (506 reasoning)
git clean -f -d src/
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 76 in → 603 out (588 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-flash:low PASS 100% 74 in → 411 out (397 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 76 in → 961 out (944 reasoning)
git -C src clean -fd
JSON Schema Structured Output
(raw) { "command": "git -C src clean -fd" }
deepseek/deepseek-v4-flash:medium PASS 100% 74 in → 441 out (433 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 74 in → 405 out (376 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
deepseek/deepseek-v4-flash:none PASS 100% 74 in → 7 out (0 reasoning)
git clean -fd src/
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 74 in → 14 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 153 in → 682 out (688 reasoning)
git clean -f src/
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 153 in → 554 out (537 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
deepseek/deepseek-v4-pro:high PASS 100% 74 in → 375 out (373 reasoning)
git clean -f src/
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 261 in → 339 out (327 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
deepseek/deepseek-v4-pro:low PASS 100% 74 in → 382 out (374 reasoning)
git clean -f src/
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 261 in → 601 out (588 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
deepseek/deepseek-v4-pro:medium PASS 100% 74 in → 386 out (379 reasoning)
git clean -f src
deepseek/deepseek-v4-pro:none PASS 100% 74 in → 8 out (0 reasoning)
git clean -f src/temp.py
deepseek/deepseek-v4-pro:xhigh PASS 100% 153 in → 880 out (873 reasoning)
git clean -f src
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 339 in → 1,141 out (1,128 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
google/gemini-3-flash-preview:high PASS 100% 80 in → 217 out (211 reasoning)
git clean -f src/
google/gemini-3-flash-preview:high__json_schema PASS 100% 137 in → 411 out (401 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
google/gemini-3-flash-preview:low PASS 100% 80 in → 199 out (193 reasoning)
git clean -fd src/
google/gemini-3-flash-preview:low__json_schema PASS 100% 81 in → 238 out (229 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command":"git clean -f src"}
google/gemini-3-flash-preview:medium PASS 100% 80 in → 474 out (468 reasoning)
git clean -f src/
google/gemini-3-flash-preview:medium__json_schema PASS 100% 137 in → 288 out (279 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command":"git clean -f src"}
google/gemini-3-flash-preview:xhigh PASS 100% 80 in → 267 out (261 reasoning)
git clean -f src/
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 137 in → 426 out (415 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 81 in → 784 out (778 reasoning)
git clean -fd src/
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 81 in → 577 out (563 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 80 in → 127 out (121 reasoning)
git clean -f src/
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 137 in → 129 out (114 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 80 in → 317 out (312 reasoning)
git clean -f src
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 137 in → 304 out (294 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 81 in → 715 out (709 reasoning)
git clean -fd src/
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 81 in → 357 out (342 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemini-3.1-pro-preview:high PASS 100% 80 in → 717 out (711 reasoning)
git clean -f src/
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 137 in → 90 out (80 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.1-pro-preview:low PASS 100% 80 in → 199 out (193 reasoning)
git clean -f src/
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 137 in → 271 out (260 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
google/gemini-3.1-pro-preview:medium PASS 100% 80 in → 245 out (239 reasoning)
git clean -f src/
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 137 in → 236 out (225 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 80 in → 465 out (459 reasoning)
git clean -f src/
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 137 in → 339 out (329 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash-lite:high PASS 100% 80 in → 433 out (428 reasoning)
git clean -f src
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 81 in → 563 out (554 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) {"command":"git clean -fd src"}
google/gemini-3.5-flash-lite:low PASS 100% 81 in → 5 out (0 reasoning)
git clean -f src
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 81 in → 10 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash-lite:max PASS 100% 80 in → 376 out (371 reasoning)
git clean -f src
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 137 in → 355 out (345 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash-lite:medium PASS 100% 81 in → 296 out (291 reasoning)
git clean -f src
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 137 in → 210 out (200 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 80 in → 364 out (359 reasoning)
git clean -f src
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 137 in → 434 out (424 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash:high PASS 100% 80 in → 568 out (562 reasoning)
git clean -f src/
google/gemini-3.5-flash:high__json_schema PASS 100% 137 in → 379 out (369 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.5-flash:low PASS 100% 80 in → 552 out (546 reasoning)
git clean -f src/
google/gemini-3.5-flash:low__json_schema PASS 100% 137 in → 157 out (142 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemini-3.5-flash:medium PASS 100% 80 in → 642 out (636 reasoning)
git clean -f src/
google/gemini-3.5-flash:medium__json_schema PASS 100% 137 in → 487 out (478 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command":"git clean -f src"}
google/gemini-3.5-flash:xhigh PASS 100% 80 in → 442 out (436 reasoning)
git clean -f src/
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 137 in → 384 out (374 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemini-3.6-flash:high PASS 100% 80 in → 437 out (431 reasoning)
git clean -f src/
google/gemini-3.6-flash:high__json_schema PASS 100% 81 in → 424 out (414 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
google/gemini-3.6-flash:low PASS 100% 81 in → 305 out (299 reasoning)
git clean -f src/
google/gemini-3.6-flash:low__json_schema PASS 100% 81 in → 15 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemini-3.6-flash:max PASS 100% 80 in → 481 out (475 reasoning)
git clean -f src/
google/gemini-3.6-flash:max__json_schema PASS 100% 137 in → 411 out (396 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemini-3.6-flash:medium PASS 100% 81 in → 593 out (587 reasoning)
git clean -f src/
google/gemini-3.6-flash:medium__json_schema PASS 100% 81 in → 294 out (284 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
google/gemini-3.6-flash:xhigh PASS 100% 80 in → 649 out (643 reasoning)
git clean -f src/
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 137 in → 675 out (665 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
google/gemma-4-26b-a4b-it:high PASS 100% 96 in → 983 out (870 reasoning)
git clean -fd src/
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 95 in → 612 out (527 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemma-4-26b-a4b-it:low PASS 100% 96 in → 1,076 out (972 reasoning)
git clean -f src/
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 96 in → 1,422 out (1,236 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemma-4-26b-a4b-it:medium PASS 100% 96 in → 524 out (517 reasoning)
git clean -f src/
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 95 in → 406 out (337 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
google/gemma-4-26b-a4b-it:none PASS 100% 93 in → 7 out (0 reasoning)
git clean -fd src/
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 93 in → 12 out (0 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
google/gemma-4-26b-a4b-it:xhigh PASS 100% 96 in → 1,069 out (912 reasoning)
git clean -f src/
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 96 in → 670 out (561 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
google/gemma-4-31b-it:high PASS 100% 96 in → 242 out (209 reasoning)
git clean -fd src
google/gemma-4-31b-it:high__json_schema PASS 100% 96 in → 290 out (236 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) { "command": "git clean -fd src" }
google/gemma-4-31b-it:low PASS 100% 96 in → 307 out (272 reasoning)
git clean -fd src
google/gemma-4-31b-it:low__json_schema PASS 100% 96 in → 400 out (1 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
google/gemma-4-31b-it:medium PASS 100% 99 in → 240 out (234 reasoning)
git clean -fd src
google/gemma-4-31b-it:medium__json_schema PASS 100% 96 in → 575 out (1 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
google/gemma-4-31b-it:none__json_schema PASS 100% 93 in → 12 out (0 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
google/gemma-4-31b-it:xhigh PASS 100% 95 in → 239 out (205 reasoning)
git clean -fd src
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 96 in → 233 out (186 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
git clean -dfx src/
JSON Schema Structured Output
(raw) { "command": "git clean -dfx src/" }
inclusionai/ling-2.6-flash PASS 100% 93 in → 11 out
``` git clean -f src/ ```
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
minimax/minimax-m2.5:high PASS 100% 104 in → 177 out (191 reasoning)
git clean -fd src/
minimax/minimax-m2.5:high__json_schema PASS 100% 104 in → 346 out (347 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd -- src/" }
minimax/minimax-m2.5:low PASS 100% 118 in → 132 out (124 reasoning)
git clean -fd src/
minimax/minimax-m2.5:low__json_schema PASS 100% 104 in → 135 out (131 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
minimax/minimax-m2.5:medium PASS 100% 104 in → 190 out (200 reasoning)
git clean -fd src/
minimax/minimax-m2.5:medium__json_schema PASS 100% 118 in → 94 out (80 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
minimax/minimax-m2.5:xhigh PASS 100% 104 in → 250 out (240 reasoning)
git clean -fd src/
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 104 in → 540 out (601 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd -- src/" }
minimax/minimax-m2.7:high PASS 100% 107 in → 579 out (572 reasoning)
git clean -f src/
minimax/minimax-m2.7:low PASS 100% 107 in → 802 out (795 reasoning)
git clean -f src/
minimax/minimax-m2.7:low__json_schema PASS 100% 236 in → 656 out (644 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
minimax/minimax-m2.7:medium PASS 100% 104 in → 1,207 out (1,324 reasoning)
git clean -f src
minimax/minimax-m2.7:medium__json_schema PASS 100% 236 in → 378 out (365 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
minimax/minimax-m2.7:xhigh PASS 100% 104 in → 793 out (858 reasoning)
git clean -fd src/
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 236 in → 536 out (524 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
minimax/minimax-m3:high PASS 100% 239 in → 91 out (87 reasoning)
git clean -f src/
minimax/minimax-m3:high__json_schema PASS 100% 190 in → 14 out (13 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
minimax/minimax-m3:low PASS 100% 242 in → 173 out (177 reasoning)
git clean -f src/
minimax/minimax-m3:low__json_schema PASS 100% 229 in → 14 out (0 reasoning)
git clean -f src/temp.py
JSON Schema Structured Output
(raw) {"command": "git clean -f src/temp.py"}
minimax/minimax-m3:medium PASS 100% 242 in → 137 out (130 reasoning)
git clean -f src/
minimax/minimax-m3:medium__json_schema PASS 100% 229 in → 14 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
minimax/minimax-m3:xhigh PASS 100% 242 in → 277 out (282 reasoning)
git clean -f src/
minimax/minimax-m3:xhigh__json_schema PASS 100% 229 in → 13 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
mistralai/devstral-2512 PASS 100% 73 in → 14 out
```bash git clean -f -d -- src/ ```
mistralai/devstral-2512__json_schema PASS 100% 73 in → 15 out
git clean -f -d -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -d -- src/"}
mistralai/mistral-medium-3-5:high PASS 100% 85 in → 6,659 out (7,291 reasoning)
git clean -f -- src/
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 73 in → 756 out (788 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd -- src/"}
mistralai/mistral-medium-3-5:low PASS 100% 85 in → 2,817 out (2,877 reasoning)
git clean -f -- src/
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 73 in → 696 out (691 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src/"}
mistralai/mistral-medium-3-5:medium PASS 100% 85 in → 2,733 out (2,899 reasoning)
git clean -f -- src/
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 73 in → 728 out (727 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src"}
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 73 in → 13 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src/"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 85 in → 399 out (404 reasoning)
git clean -f -- src/
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 73 in → 571 out (571 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src/"}
moonshotai/kimi-k2.6:high PASS 100% 76 in → 932 out (995 reasoning)
git clean -f src/
moonshotai/kimi-k2.6:high__json_schema PASS 100% 77 in → 647 out (670 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
moonshotai/kimi-k2.6:low PASS 100% 77 in → 699 out (691 reasoning)
git clean -f src/
moonshotai/kimi-k2.6:low__json_schema PASS 100% 76 in → 1,360 out (1,166 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
moonshotai/kimi-k2.6:medium PASS 100% 77 in → 1,484 out (1,477 reasoning)
git clean -f src/
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 76 in → 1,875 out (1,682 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
moonshotai/kimi-k2.6:xhigh PASS 100% 77 in → 1,007 out (1,071 reasoning)
git clean -f src/
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 259 in → 1,377 out (1,365 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
moonshotai/kimi-k2.7-code:high PASS 100% 77 in → 145 out (149 reasoning)
git clean -fd src
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 77 in → 241 out (234 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
moonshotai/kimi-k2.7-code:low PASS 100% 77 in → 394 out (403 reasoning)
git clean -f src
moonshotai/kimi-k2.7-code:medium PASS 100% 77 in → 382 out (401 reasoning)
git clean -f src/
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 77 in → 1,092 out (1,080 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command": "git clean -f src"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 77 in → 391 out (397 reasoning)
git clean -f src
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 160 in → 571 out (558 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
moonshotai/kimi-k3:high PASS 100% 154 in → 174 out (153 reasoning)
git clean -f src/
moonshotai/kimi-k3:high__json_schema PASS 100% 250 in → 255 out (230 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
moonshotai/kimi-k3:low PASS 100% 154 in → 458 out (437 reasoning)
git clean -fd src/
moonshotai/kimi-k3:low__json_schema PASS 100% 154 in → 240 out (219 reasoning)
git clean -f src/
moonshotai/kimi-k3:max PASS 100% 154 in → 194 out (172 reasoning)
git clean -f -- src/
moonshotai/kimi-k3:max__json_schema PASS 100% 154 in → 325 out (304 reasoning)
git clean -f src/
moonshotai/kimi-k3:medium PASS 100% 154 in → 311 out (290 reasoning)
git clean -f src/
moonshotai/kimi-k3:medium__json_schema PASS 100% 154 in → 181 out (160 reasoning)
git clean -f src/
moonshotai/kimi-k3:xhigh PASS 100% 154 in → 181 out (160 reasoning)
git clean -f src/
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 154 in → 266 out (245 reasoning)
git clean -f src/
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 86 in → 757 out (805 reasoning)
git clean -f -- src/
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 86 in → 475 out (475 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 86 in → 425 out (436 reasoning)
git clean -f -- src/
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 86 in → 514 out (533 reasoning)
git clean -fd -- 'src'
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 86 in → 579 out (610 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src/" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 86 in → 471 out (491 reasoning)
git clean -fd src/
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 86 in → 426 out (436 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 86 in → 136 out (139 reasoning)
git clean -f src
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 86 in → 209 out (183 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 86 in → 223 out (202 reasoning)
git clean -f -- src/
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 86 in → 578 out (541 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 86 in → 134 out (120 reasoning)
git clean -f src/
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 86 in → 258 out (234 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 86 in → 7 out (0 reasoning)
git clean -f src/
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 86 in → 14 out (0 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 86 in → 221 out (219 reasoning)
git clean -f src
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 86 in → 133 out (115 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
openai/gpt-5.4-mini:high PASS 100% 69 in → 205 out (192 reasoning)
git clean -f -- src/
openai/gpt-5.4-mini:high__json_schema PASS 100% 104 in → 183 out (162 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src/"}
openai/gpt-5.4-mini:low PASS 100% 69 in → 98 out (85 reasoning)
git clean -f -- src/
openai/gpt-5.4-mini:low__json_schema PASS 100% 104 in → 88 out (67 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.4-mini:medium PASS 100% 69 in → 140 out (127 reasoning)
git clean -f -- src/
openai/gpt-5.4-mini:medium__json_schema PASS 100% 104 in → 167 out (146 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src/"}
openai/gpt-5.4-mini:none PASS 100% 69 in → 10 out (0 reasoning)
git clean -f src/
openai/gpt-5.4-mini:none__json_schema PASS 100% 104 in → 24 out (0 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.4-mini:xhigh PASS 100% 69 in → 1,273 out (1,260 reasoning)
git clean -f -- src/
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 104 in → 1,368 out (1,347 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.4-nano:high PASS 100% 69 in → 10 out (0 reasoning)
git clean -fd -- src
openai/gpt-5.4-nano:high__json_schema PASS 100% 104 in → 87 out (66 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src/"}
openai/gpt-5.4-nano:low PASS 100% 69 in → 10 out (0 reasoning)
git clean -fd -- src
openai/gpt-5.4-nano:low__json_schema PASS 100% 104 in → 18 out (0 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.4-nano:medium__json_schema PASS 100% 104 in → 18 out (0 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.4-nano:none PASS 100% 69 in → 19 out (0 reasoning)
git clean -nd -- src/ && git clean -fd -- src/
openai/gpt-5.4-nano:none__json_schema PASS 100% 104 in → 25 out (0 reasoning)
git clean -nd -- src && git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -nd -- src && git clean -fd -- src"}
openai/gpt-5.4-nano:xhigh PASS 100% 69 in → 223 out (210 reasoning)
git clean -fd -- src/
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 104 in → 895 out (874 reasoning)
git clean -fd -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src/"}
openai/gpt-5.4:high PASS 100% 69 in → 175 out (162 reasoning)
git clean -f -- src/
openai/gpt-5.4:high__json_schema PASS 100% 104 in → 163 out (143 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.4:low PASS 100% 69 in → 49 out (37 reasoning)
git clean -f -- src
openai/gpt-5.4:low__json_schema PASS 100% 104 in → 70 out (50 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src"}
openai/gpt-5.4:medium PASS 100% 69 in → 93 out (81 reasoning)
git clean -f -- src
openai/gpt-5.4:medium__json_schema PASS 100% 104 in → 161 out (141 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.4:none PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.4:none__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.4:xhigh PASS 100% 69 in → 423 out (410 reasoning)
git clean -f -- src/
openai/gpt-5.4:xhigh__json_schema PASS 100% 104 in → 537 out (516 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.5:high PASS 100% 69 in → 392 out (379 reasoning)
git clean -f -- src/
openai/gpt-5.5:high__json_schema PASS 100% 104 in → 250 out (229 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.5:low PASS 100% 69 in → 171 out (158 reasoning)
git clean -f -- src/
openai/gpt-5.5:low__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.5:medium PASS 100% 69 in → 121 out (108 reasoning)
git clean -f -- src/
openai/gpt-5.5:medium__json_schema PASS 100% 104 in → 496 out (475 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.5:none PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.5:none__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.5:xhigh PASS 100% 69 in → 298 out (285 reasoning)
git clean -f -- src/
openai/gpt-5.5:xhigh__json_schema PASS 100% 104 in → 436 out (415 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-luna:high PASS 100% 69 in → 99 out (86 reasoning)
git clean -f -- src/
openai/gpt-5.6-luna:high__json_schema PASS 100% 104 in → 214 out (194 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.6-luna:low PASS 100% 69 in → 55 out (43 reasoning)
git clean -f -- src
openai/gpt-5.6-luna:low__json_schema PASS 100% 104 in → 52 out (32 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.6-luna:max PASS 100% 69 in → 144 out (131 reasoning)
git clean -f -- src/
openai/gpt-5.6-luna:max__json_schema PASS 100% 104 in → 230 out (209 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-luna:medium PASS 100% 69 in → 110 out (97 reasoning)
git clean -fd -- src/
openai/gpt-5.6-luna:medium__json_schema PASS 100% 104 in → 110 out (89 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-luna:none PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-luna:none__json_schema PASS 100% 104 in → 18 out (0 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-5.6-luna:xhigh PASS 100% 69 in → 112 out (99 reasoning)
git clean -f -- src/
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 104 in → 183 out (162 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:high PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:high__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:low PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:low__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:max PASS 100% 69 in → 166 out (153 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:max__json_schema PASS 100% 104 in → 247 out (226 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:medium PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:medium__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:none PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:none__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-sol:xhigh PASS 100% 69 in → 48 out (35 reasoning)
git clean -f -- src/
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:high PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:high__json_schema PASS 100% 104 in → 55 out (34 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:low PASS 100% 69 in → 50 out (37 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:low__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:max PASS 100% 69 in → 187 out (174 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:max__json_schema PASS 100% 104 in → 133 out (112 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:medium PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:medium__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:none PASS 100% 69 in → 11 out (0 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:none__json_schema PASS 100% 104 in → 19 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-5.6-terra:xhigh PASS 100% 69 in → 99 out (86 reasoning)
git clean -f -- src/
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 104 in → 105 out (84 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
openai/gpt-oss-120b:high PASS 100% 117 in → 676 out (722 reasoning)
git clean -f src
openai/gpt-oss-120b:high__json_schema PASS 100% 132 in → 513 out (516 reasoning)
git clean -fd -- src
JSON Schema Structured Output
(raw) {"command":"git clean -fd -- src"}
openai/gpt-oss-120b:low PASS 100% 130 in → 48 out (34 reasoning)
git clean -fd src/
openai/gpt-oss-120b:low__json_schema PASS 100% 130 in → 67 out (45 reasoning)
git clean -f -d src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -d src/"}
openai/gpt-oss-120b:medium PASS 100% 128 in → 145 out (134 reasoning)
git clean -f src/
openai/gpt-oss-120b:medium__json_schema PASS 100% 128 in → 170 out (155 reasoning)
git clean -f -d -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -d -- src/"}
openai/gpt-oss-120b:xhigh PASS 100% 128 in → 430 out (451 reasoning)
git clean -f src
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 130 in → 515 out (502 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) {"command":"git clean -fd src"}
openai/gpt-oss-20b:high PASS 100% 130 in → 169 out (164 reasoning)
git clean -fd src
openai/gpt-oss-20b:high__json_schema PASS 100% 128 in → 794 out (798 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command":"git clean -fd src/"}
openai/gpt-oss-20b:low PASS 100% 130 in → 97 out (71 reasoning)
git -C src clean -fd
openai/gpt-oss-20b:medium PASS 100% 130 in → 524 out (517 reasoning)
git clean -f -d src
openai/gpt-oss-20b:medium__json_schema PASS 100% 128 in → 237 out (211 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) { "command": "git clean -fd src" }
openai/gpt-oss-20b:xhigh PASS 100% 130 in → 257 out (250 reasoning)
git clean -f -d src
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 132 in → 2,474 out (2,756 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) {"command":"git clean -f src"}
poolside/laguna-m.1:high PASS 100% 83 in → 597 out (588 reasoning)
git clean -fd src/
poolside/laguna-m.1:high__json_schema PASS 100% 83 in → 462 out (445 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
poolside/laguna-m.1:low PASS 100% 83 in → 473 out (464 reasoning)
git clean -fd src/
poolside/laguna-m.1:low__json_schema PASS 100% 83 in → 494 out (477 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
poolside/laguna-m.1:medium PASS 100% 83 in → 879 out (870 reasoning)
git clean -f src/
poolside/laguna-m.1:medium__json_schema PASS 100% 83 in → 2,229 out (2,213 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
poolside/laguna-m.1:none PASS 100% 83 in → 8 out (0 reasoning)
git clean -fd src/
poolside/laguna-m.1:none__json_schema PASS 100% 83 in → 12 out (0 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
poolside/laguna-m.1:xhigh PASS 100% 83 in → 1,411 out (1,402 reasoning)
git clean -f src/
poolside/laguna-m.1:xhigh__json_schema PASS 100% 83 in → 473 out (460 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) {"command": "git clean -fd src/"}
poolside/laguna-xs-2.1:high PASS 100% 83 in → 478 out (472 reasoning)
git clean -f src
poolside/laguna-xs-2.1:high__json_schema PASS 100% 83 in → 299 out (283 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) { "command": "git clean -fd src" }
poolside/laguna-xs-2.1:low__json_schema PASS 100% 83 in → 444 out (431 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src"}
poolside/laguna-xs-2.1:medium PASS 100% 83 in → 663 out (656 reasoning)
git clean -f src/
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 83 in → 678 out (666 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) {"command": "git clean -fd src"}
poolside/laguna-xs-2.1:none PASS 100% 83 in → 6 out (0 reasoning)
git clean -f src/
poolside/laguna-xs-2.1:none__json_schema PASS 100% 83 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
poolside/laguna-xs-2.1:xhigh PASS 100% 83 in → 1,486 out (1,479 reasoning)
git clean -f src/
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 83 in → 804 out (787 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
poolside/laguna-xs.2:high PASS 100% 120 in → 281 out (272 reasoning)
git clean -f src/
poolside/laguna-xs.2:high__json_schema PASS 100% 120 in → 179 out (162 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
poolside/laguna-xs.2:low PASS 100% 120 in → 206 out (198 reasoning)
git clean -f src
poolside/laguna-xs.2:low__json_schema PASS 100% 120 in → 145 out (128 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
poolside/laguna-xs.2:medium PASS 100% 120 in → 127 out (118 reasoning)
git clean -f src/
poolside/laguna-xs.2:medium__json_schema PASS 100% 120 in → 269 out (252 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
poolside/laguna-xs.2:none PASS 100% 120 in → 8 out (0 reasoning)
git clean -f src/
poolside/laguna-xs.2:none__json_schema PASS 100% 120 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
poolside/laguna-xs.2:xhigh PASS 100% 120 in → 126 out (118 reasoning)
git clean -fd src
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 120 in → 148 out (135 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
qwen/qwen3.6-27b:high PASS 100% 79 in → 1,547 out (1,402 reasoning)
git clean -f src/
qwen/qwen3.6-27b:high__json_schema PASS 100% 81 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-27b:low PASS 100% 79 in → 1,357 out (1,269 reasoning)
git clean -f src/
qwen/qwen3.6-27b:low__json_schema PASS 100% 79 in → 1,688 out (1,526 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-27b:medium PASS 100% 79 in → 2,088 out (1,799 reasoning)
git clean -f src/
qwen/qwen3.6-27b:none PASS 100% 81 in → 7 out (0 reasoning)
git clean -f src/
qwen/qwen3.6-27b:none__json_schema PASS 100% 81 in → 16 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-27b:xhigh PASS 100% 79 in → 1,365 out (1,307 reasoning)
git clean -df src
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 79 in → 1,581 out (1,366 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-35b-a3b:high PASS 100% 79 in → 1,398 out (1,387 reasoning)
git clean -f src/
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 79 in → 1,827 out (1,648 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-35b-a3b:low PASS 100% 79 in → 1,391 out (1,287 reasoning)
git clean -f src/
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 79 in → 2,175 out (1,902 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 79 in → 1,927 out (1,916 reasoning)
git clean -f src/
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 79 in → 1,417 out (1,275 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-35b-a3b:none PASS 100% 81 in → 5 out (0 reasoning)
git clean -fd src
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 81 in → 17 out (0 reasoning)
git clean -fdx src
JSON Schema Structured Output
(raw) { "command": "git clean -fdx src" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 79 in → 1,443 out (1,290 reasoning)
git clean -f src/
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 79 in → 1,746 out (1,593 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
qwen/qwen3.6-flash:high PASS 100% 79 in → 1,734 out (1,724 reasoning)
git clean -f src
qwen/qwen3.6-flash:low PASS 100% 79 in → 1,301 out (1,290 reasoning)
git clean -f src/
qwen/qwen3.6-flash:medium PASS 100% 79 in → 1,353 out (1,342 reasoning)
git clean -f src/
qwen/qwen3.6-flash:none PASS 100% 81 in → 5 out (0 reasoning)
git clean -fd src
qwen/qwen3.6-flash:xhigh PASS 100% 79 in → 2,116 out (2,105 reasoning)
git clean -f src/
qwen/qwen3.7-flash:high PASS 100% 79 in → 1,324 out (1,313 reasoning)
git clean -f src/
qwen/qwen3.7-flash:low PASS 100% 79 in → 1,496 out (1,485 reasoning)
git clean -f src/
qwen/qwen3.7-flash:max PASS 100% 79 in → 1,917 out (1,906 reasoning)
git clean -f src/
qwen/qwen3.7-flash:medium PASS 100% 79 in → 1,461 out (1,449 reasoning)
git clean -f -- src/
qwen/qwen3.7-flash:none PASS 100% 81 in → 5 out (0 reasoning)
git clean -fd src
qwen/qwen3.7-flash:xhigh PASS 100% 79 in → 1,714 out (1,703 reasoning)
git clean -f src/
qwen/qwen3.7-max:high PASS 100% 79 in → 1,008 out (998 reasoning)
git clean -f src
qwen/qwen3.7-max:low PASS 100% 79 in → 1,107 out (1,096 reasoning)
git clean -f src/
qwen/qwen3.7-max:medium PASS 100% 79 in → 897 out (886 reasoning)
git clean -f src/
qwen/qwen3.7-max:none PASS 100% 81 in → 6 out (0 reasoning)
git clean -f src/
qwen/qwen3.7-max:xhigh PASS 100% 79 in → 1,087 out (1,077 reasoning)
git clean -f src
qwen/qwen3.7-plus:high PASS 100% 79 in → 803 out (793 reasoning)
git clean -f src
qwen/qwen3.7-plus:low PASS 100% 79 in → 939 out (928 reasoning)
git clean -f src/
qwen/qwen3.7-plus:medium PASS 100% 79 in → 970 out (960 reasoning)
git clean -f src
qwen/qwen3.7-plus:none PASS 100% 81 in → 6 out (0 reasoning)
git clean -f src/
qwen/qwen3.7-plus:xhigh PASS 100% 79 in → 1,003 out (993 reasoning)
git clean -f src
tencent/hy3:high PASS 100% 79 in → 1,906 out (1,898 reasoning)
git clean -f -- src
tencent/hy3:high__json_schema PASS 100% 79 in → 1,223 out (1,101 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
tencent/hy3:low PASS 100% 79 in → 1,208 out (1,200 reasoning)
git clean -f -- src
tencent/hy3:low__json_schema PASS 100% 79 in → 1,009 out (981 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src" }
tencent/hy3:medium PASS 100% 79 in → 933 out (925 reasoning)
git clean -f -- src
tencent/hy3:medium__json_schema PASS 100% 79 in → 915 out (889 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
tencent/hy3:none PASS 100% 82 in → 6 out (0 reasoning)
git clean -f src
tencent/hy3:none__json_schema PASS 100% 82 in → 16 out (0 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
tencent/hy3:xhigh PASS 100% 79 in → 1,576 out (1,568 reasoning)
git clean -f -- src
tencent/hy3:xhigh__json_schema PASS 100% 79 in → 1,769 out (1,681 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src" }
thinkingmachines/inkling-small:high PASS 100% 78 in → 209 out (219 reasoning)
git clean -f src
thinkingmachines/inkling-small:high__json_schema PASS 100% 78 in → 187 out (169 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
thinkingmachines/inkling-small:low PASS 100% 78 in → 73 out (63 reasoning)
git clean -fd src
thinkingmachines/inkling-small:low__json_schema PASS 100% 78 in → 55 out (40 reasoning)
git clean -fd src
JSON Schema Structured Output
(raw) {"command": "git clean -fd src"}
thinkingmachines/inkling-small:max PASS 100% 78 in → 293 out (281 reasoning)
git clean -f src/
thinkingmachines/inkling-small:max__json_schema PASS 100% 78 in → 277 out (259 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
thinkingmachines/inkling-small:medium PASS 100% 78 in → 156 out (154 reasoning)
git clean -f src
thinkingmachines/inkling-small:medium__json_schema PASS 100% 78 in → 86 out (69 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src" }
thinkingmachines/inkling-small:none PASS 100% 76 in → 10 out (0 reasoning)
git clean -f -d src
thinkingmachines/inkling-small:xhigh PASS 100% 78 in → 235 out (245 reasoning)
git clean -f src/
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 78 in → 1,161 out (1,143 reasoning)
git clean -f -- src
JSON Schema Structured Output
(raw) { "command": "git clean -f -- src" }
thinkingmachines/inkling:high PASS 100% 78 in → 105 out (93 reasoning)
git clean -fd src/
thinkingmachines/inkling:high__json_schema PASS 100% 78 in → 1,529 out (1,512 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
thinkingmachines/inkling:low PASS 100% 78 in → 42 out (30 reasoning)
git clean -fd -- src
thinkingmachines/inkling:low__json_schema PASS 100% 78 in → 68 out (50 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
thinkingmachines/inkling:max PASS 100% 78 in → 434 out (422 reasoning)
git clean -f src/
thinkingmachines/inkling:medium PASS 100% 78 in → 101 out (90 reasoning)
git clean -fd src
thinkingmachines/inkling:xhigh PASS 100% 78 in → 417 out (404 reasoning)
git clean -f src/
thinkingmachines/inkling:xhigh__json_schema PASS 100% 78 in → 290 out (273 reasoning)
git clean -fd src/
JSON Schema Structured Output
(raw) { "command": "git clean -fd src/" }
x-ai/grok-4.3:high PASS 100% 248 in → 473 out (467 reasoning)
git clean -f src/
x-ai/grok-4.3:high__json_schema PASS 100% 304 in → 712 out (703 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
x-ai/grok-4.3:low PASS 100% 254 in → 336 out (330 reasoning)
git clean -f src/
x-ai/grok-4.3:low__json_schema PASS 100% 310 in → 395 out (386 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
x-ai/grok-4.3:max PASS 100% 248 in → 280 out (274 reasoning)
git clean -f src/
x-ai/grok-4.3:max__json_schema PASS 100% 304 in → 576 out (567 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
x-ai/grok-4.3:medium PASS 100% 254 in → 520 out (514 reasoning)
git clean -f src/
x-ai/grok-4.3:medium__json_schema PASS 100% 310 in → 592 out (579 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
x-ai/grok-4.3:none PASS 100% 246 in → 6 out (0 reasoning)
git clean -f src/
x-ai/grok-4.3:none__json_schema PASS 100% 302 in → 10 out (0 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
x-ai/grok-4.3:xhigh PASS 100% 248 in → 394 out (387 reasoning)
`git clean -f src/`
x-ai/grok-4.3:xhigh__json_schema PASS 100% 304 in → 571 out (562 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
x-ai/grok-4.5:high PASS 100% 272 in → 725 out (718 reasoning)
git clean -f -- src/
x-ai/grok-4.5:high__json_schema PASS 100% 338 in → 673 out (661 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command":"git clean -f -- src/"}
x-ai/grok-4.5:low PASS 100% 272 in → 390 out (381 reasoning)
``` git clean -f src/ ```
x-ai/grok-4.5:low__json_schema PASS 100% 338 in → 332 out (321 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
x-ai/grok-4.5:max PASS 100% 272 in → 445 out (439 reasoning)
git clean -f src/
x-ai/grok-4.5:max__json_schema PASS 100% 338 in → 675 out (664 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
x-ai/grok-4.5:medium PASS 100% 272 in → 403 out (397 reasoning)
git clean -f src/
x-ai/grok-4.5:medium__json_schema PASS 100% 338 in → 540 out (528 reasoning)
git clean -f -- src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -- src/"}
x-ai/grok-4.5:xhigh PASS 100% 272 in → 543 out (538 reasoning)
git clean -f src
x-ai/grok-4.5:xhigh__json_schema PASS 100% 338 in → 489 out (478 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command":"git clean -f src/"}
z-ai/glm-4.7-flash:high PASS 100% 69 in → 1,252 out (1,251 reasoning)
git clean -fd src
z-ai/glm-4.7-flash:low PASS 100% 69 in → 1,486 out (1,459 reasoning)
git clean -d -f src/
z-ai/glm-4.7-flash:low__json_schema PASS 100% 69 in → 707 out (651 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
z-ai/glm-4.7-flash:medium PASS 100% 69 in → 832 out (801 reasoning)
git clean -f src/
z-ai/glm-4.7-flash:none PASS 100% 69 in → 9 out (0 reasoning)
git clean -f -d src/
z-ai/glm-4.7-flash:none__json_schema PASS 100% 69 in → 21 out (0 reasoning)
git clean -f -d -e root.txt -e test/
JSON Schema Structured Output
(raw) { "command": "git clean -f -d -e root.txt -e test/" }
z-ai/glm-4.7-flash:xhigh PASS 100% 69 in → 509 out (479 reasoning)
git clean -f -d src
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 69 in → 786 out (723 reasoning)
git clean -f -d src/
JSON Schema Structured Output
(raw) {"command": "git clean -f -d src/"}
z-ai/glm-5.2:high PASS 100% 76 in → 531 out (523 reasoning)
git clean -f src/
z-ai/glm-5.2:high__json_schema PASS 100% 69 in → 641 out (573 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) {"command": "git clean -f src/"}
z-ai/glm-5.2:low PASS 100% 76 in → 669 out (661 reasoning)
git clean -f src/
z-ai/glm-5.2:low__json_schema PASS 100% 69 in → 508 out (447 reasoning)
git clean -f src
JSON Schema Structured Output
(raw) { "command": "git clean -f src"}
z-ai/glm-5.2:medium PASS 100% 76 in → 516 out (508 reasoning)
git clean -f src/
z-ai/glm-5.2:medium__json_schema PASS 100% 69 in → 617 out (557 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
z-ai/glm-5.2:none PASS 100% 70 in → 6 out (0 reasoning)
git clean -f src/
z-ai/glm-5.2:none__json_schema PASS 100% 69 in → 15 out (0 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
z-ai/glm-5.2:xhigh PASS 100% 76 in → 1,022 out (1,014 reasoning)
git clean -f src/
z-ai/glm-5.2:xhigh__json_schema PASS 100% 76 in → 14 out (13 reasoning)
git clean -f src/
JSON Schema Structured Output
(raw) { "command": "git clean -f src/" }
arcee-ai/trinity-mini:high FAIL 75% 73 in → 398 out (415 reasoning)
gitclean -f src/temp.py
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
arcee-ai/trinity-mini:medium FAIL 75% 73 in → 365 out (408 reasoning)
git rmsrc/temp.py
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 75% 73 in → 375 out (402 reasoning)
git clean -fsrc
JSON Schema Structured Output
(raw) { "command": "git clean -fsrc" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
arcee-ai/trinity-mini:xhigh FAIL 75% 73 in → 386 out (431 reasoning)
gitclean -fd src
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
deepseek/deepseek-v4-pro:medium__json_schema FAIL 75% 76 in → 370 out (353 reasoning)
git clean -n src/temp.py
JSON Schema Structured Output
(raw) { "command": "git clean -n src/temp.py" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
deepseek/deepseek-v4-pro:none__json_schema FAIL 75% 74 in → 11 out (0 reasoning)
git clean -n src/
JSON Schema Structured Output
(raw) {"command":"git clean -n src/"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
ibm-granite/granite-4.1-8b FAIL 75% 88 in → 11 out
git clean -fd -X -d src/
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
liquid/lfm-2-24b-a2b FAIL 75% 82 in → 50 out
git rm --ignore-unmatch -r src/* --exclude-dir=test src/temp.py # Explanation: The command removes untracked files under src but skips others like root.txt and files in test/
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
mistralai/mistral-medium-3-5:none FAIL 75% 85 in → 16 out (0 reasoning)
git clean -f --directory -e root.txt -e test src/
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 75% 86 in → 165 out (136 reasoning)
git rm -r --cached --ignore-unmatch src/temp.py src/test/temp.py
JSON Schema Structured Output
(raw) {"command":"git rm -r --cached --ignore-unmatch src/temp.py src/test/temp.py"}
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 75% 86 in → 11 out (0 reasoning)
git clean -nfd -- src/temp.py
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 75% 86 in → 16 out (0 reasoning)
git clean -Xf -- src
JSON Schema Structured Output
(raw) { "command": "git clean -Xf -- src" }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
openai/gpt-5.4-nano:medium FAIL 75% 69 in → 12 out (0 reasoning)
git clean -ndf -- src/
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
openai/gpt-oss-20b:low__json_schema FAIL 75% 130 in → 92 out (0 reasoning)
//?..???..--..?????……?  …???-……5..………..?..K?…212… As ... ??………………
………​???...?… ...…….……33? ……
----..…  …… … ????…??..
JSON Schema Structured Output
(raw) { "command" : "//?..???..--..?????……?  …???-……5..………..?..K?…212… As ... ??………………
………​???...?… ...…….……33? ……
----..…  …… … ????…??.. " }
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
poolside/laguna-xs-2.1:low FAIL 75% 83 in → 217 out (217 reasoning)
(empty output)
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
thinkingmachines/inkling:none FAIL 75% 78 in → 9 out (0 reasoning)
git clean -fd src/<|end_message|>
Failure: Failed assertions: [{'type': 'file_not_exists', 'path': 'src/temp.py'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
Failure: Model 'google/gemma-4-31b-it:none' violated the reasoning_level=none invariant: provider reported 188 reasoning token(s)
Invalid JSON. Output: git rm --ignore-unmatch -r src/.* src/temp.py git rm --ignore-unmatch -r src/.* test/.*
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c11aa748-0e10-9b82-91cb-29a9d98b3da8","request_id":"c11aa748-0e10-9b82-91cb-29a9d98b3da8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a844fd74-1b60-9d9e-9977-a7b28fdeb112","request_id":"a844fd74-1b60-9d9e-9977-a7b28fdeb112"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f5377ae7-7909-9089-8a95-860e9f4db850","request_id":"f5377ae7-7909-9089-8a95-860e9f4db850"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8f9c9e8d-2cfd-929f-b4b0-e08c1f11ac8c","request_id":"8f9c9e8d-2cfd-929f-b4b0-e08c1f11ac8c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-103d3e23-d91e-9a29-b467-72adf8b66238","request_id":"103d3e23-d91e-9a29-b467-72adf8b66238"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-61c68e59-9d68-980c-9e18-7130a1c8fd87"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-ce90e09d-626e-9958-9f94-8f4ab2fe8c3d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3a492684-ff68-9d1b-bf64-1299787e4893"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-b44e0d98-e26f-9ac5-8164-3c57d5cefb20"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-58ce2e38-0ac2-9a9d-b338-34a57174d987"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7283a79d-1139-9ffe-95d1-7e7c51fc495d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7a79be56-2700-9925-b68b-b00c1fecffa5","request_id":"7a79be56-2700-9925-b68b-b00c1fecffa5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-550bd9a9-cc1d-9f0c-80c8-3f05ab6e2305","request_id":"550bd9a9-cc1d-9f0c-80c8-3f05ab6e2305"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b88c0f51-c111-95d1-b6a0-20614fb993d7","request_id":"b88c0f51-c111-95d1-b6a0-20614fb993d7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-56c6cb28-d306-9d2e-ac92-c89c4e42cc61","request_id":"56c6cb28-d306-9d2e-ac92-c89c4e42cc61"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-54f7c2c5-ee27-9501-896e-2ec1c1167253","request_id":"54f7c2c5-ee27-9501-896e-2ec1c1167253"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e2f89b7f-964d-95b4-b4c3-386c30154620","request_id":"e2f89b7f-964d-95b4-b4c3-386c30154620"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-121de262-5218-98e5-9389-ad82a8cf651d","request_id":"121de262-5218-98e5-9389-ad82a8cf651d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1489128e-2a5d-9837-9489-a50dd3b67df0","request_id":"1489128e-2a5d-9837-9489-a50dd3b67df0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-32e0b3b6-07fc-951b-a29b-c4cae0ffbef9","request_id":"32e0b3b6-07fc-951b-a29b-c4cae0ffbef9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-64967a88-a7d5-9bd2-a4ec-9a5d0600a359","request_id":"64967a88-a7d5-9bd2-a4ec-9a5d0600a359"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
Invalid JSON. Output: git clean -fd src{ "command": "git clean -fd src" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -f src/
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd src/
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git clean -fd src{"command": "git clean -fd src"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)