Create worktree and make commit
Tests ability to create a worktree and immediately make a commit in it. Evaluates end-to-end worktree commit workflow.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello' > hello.txt
  5. 05 git add hello.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 git checkout -b feature
  8. 08 echo 'Feature' > feature.txt
  9. 09 git add feature.txt
  10. 10 git commit -m 'Add feature'
  11. 11 git checkout main
Prompt
Create a worktree for 'feature' at '../feature-wt', then create a file '../feature-wt/new.txt' with content 'new file', add it and commit with message 'Add new file' in the worktree. Output ONLY the commands to run, one per line.
Expected
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'

Scoped model quality, cost, API time, and token usage for worktree_usage/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5.1:high PASS 100% 191 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-fable-5.1:low PASS 100% 200 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-fable-5.1:low__json_schema PASS 100% 430 in → 88 out (0 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5.1:max PASS 100% 202 in → 2,155 out (2,080 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-fable-5.1:max__json_schema PASS 100% 427 in → 1,447 out (1,359 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5.1:medium PASS 100% 202 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-fable-5.1:medium__json_schema PASS 100% 434 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5.1:xhigh PASS 100% 196 in → 569 out (477 reasoning)
git worktree add ../feature-wt feature || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-fable-5.1:xhigh__json_schema PASS 100% 434 in → 1,163 out (1,050 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:high__json_schema PASS 100% 429 in → 110 out (34 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:low__json_schema PASS 100% 431 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:max__json_schema PASS 100% 433 in → 825 out (101 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:medium__json_schema PASS 100% 433 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 432 in → 485 out (96 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:high PASS 100% 195 in → 412 out (277 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 381 in → 3,940 out (1,783 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:low PASS 100% 197 in → 735 out (549 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 384 in → 1,276 out (626 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:medium PASS 100% 198 in → 572 out (403 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 384 in → 1,009 out (646 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:none PASS 100% 168 in → 66 out (0 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 351 in → 86 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt status sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "git -C ../feature-wt status", "sh -c 'echo \"new file\" > ../feature-wt/new.txt'", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 384 in → 2,630 out (1,256 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:high PASS 100% 167 in → 120 out (37 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:low PASS 100% 173 in → 64 out (0 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 352 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 355 in → 99 out (15 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:none PASS 100% 168 in → 64 out (0 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 358 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 172 in → 157 out (65 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.7:high PASS 100% 203 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
anthropic/claude-opus-4.7:high__json_schema PASS 100% 437 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:low PASS 100% 201 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:low__json_schema PASS 100% 431 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:max__json_schema PASS 100% 432 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:medium PASS 100% 202 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 436 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c "echo 'new file' > ../feature-wt/new.txt" git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c \"echo 'new file' > ../feature-wt/new.txt\"","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:none PASS 100% 203 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
anthropic/claude-opus-4.7:none__json_schema PASS 100% 428 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c "echo 'new file' > ../feature-wt/new.txt" git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c \"echo 'new file' > ../feature-wt/new.txt\"","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 202 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 431 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 428 in → 124 out (42 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:low__json_schema PASS 100% 433 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 431 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 428 in → 83 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 435 in → 304 out (43 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5.5:high PASS 100% 198 in → 311 out (236 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-5.5:high__json_schema PASS 100% 433 in → 713 out (629 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5.5:low PASS 100% 197 in → 319 out (244 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-5.5:low__json_schema PASS 100% 430 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5.5:max PASS 100% 198 in → 7,593 out (7,493 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-5.5:max__json_schema PASS 100% 433 in → 6,837 out (6,736 reasoning)
git worktree add ../feature-wt feature || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature || git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5.5:medium PASS 100% 198 in → 378 out (303 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-5.5:medium__json_schema PASS 100% 431 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5.5:xhigh PASS 100% 199 in → 443 out (368 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-5.5:xhigh__json_schema PASS 100% 435 in → 515 out (431 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5:high__json_schema PASS 100% 428 in → 179 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5:low PASS 100% 194 in → 87 out (0 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-opus-5:low__json_schema PASS 100% 430 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5:max__json_schema PASS 100% 431 in → 555 out (58 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5:medium__json_schema PASS 100% 426 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-5:xhigh__json_schema PASS 100% 428 in → 163 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:low PASS 100% 168 in → 104 out (9 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 351 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 166 in → 150 out (60 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 355 in → 120 out (34 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:none PASS 100% 174 in → 66 out (0 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 354 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-5:high__json_schema PASS 100% 433 in → 237 out (59 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 435 in → 166 out (38 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
arcee-ai/trinity-large-thinking:xhigh PASS 100% 142 in → 1,403 out (1,364 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
deepseek/deepseek-v4-flash:low PASS 100% 145 in → 530 out (485 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 251 in → 56 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 146 in → 68 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 148 in → 69 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 343 in → 585 out (528 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
deepseek/deepseek-v4.1-flash:max PASS 100% 168 in → 7,055 out (7,004 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
deepseek/deepseek-v4.1-flash:max__json_schema PASS 100% 275 in → 519 out (461 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3-flash-preview:low__json_schema PASS 100% 247 in → 770 out (710 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 172 in → 179 out (127 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 243 in → 170 out (114 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 171 in → 616 out (564 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 174 in → 376 out (364 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 169 in → 920 out (888 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-pro-preview:high PASS 100% 168 in → 2,569 out (2,517 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 249 in → 932 out (874 reasoning)
git worktree add -B feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -B feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 247 in → 4,048 out (3,992 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 248 in → 1,622 out (1,566 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.5-flash-lite:low PASS 100% 172 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 168 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
google/gemini-3.5-flash-lite:max PASS 100% 166 in → 1,953 out (1,901 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 243 in → 1,329 out (1,273 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.5-flash-lite:medium PASS 100% 170 in → 696 out (644 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 171 in → 451 out (419 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.6-flash:max PASS 100% 167 in → 1,802 out (1,750 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.8-flash:high__json_schema PASS 100% 248 in → 10,907 out (10,831 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemini-3.8-flash:max PASS 100% 167 in → 14,408 out (14,356 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.8-flash:max__json_schema PASS 100% 252 in → 14,345 out (14,290 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.8-flash:medium PASS 100% 162 in → 5,109 out (5,052 reasoning)
```bash git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
google/gemini-3.8-flash:medium__json_schema PASS 100% 168 in → 10,938 out (10,894 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemini-3.8-flash:xhigh PASS 100% 166 in → 31,849 out (31,795 reasoning)
git worktree add -B feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.8-flash:xhigh__json_schema PASS 100% 251 in → 13,408 out (13,353 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemma-4-26b-a4b-it:medium PASS 100% 191 in → 3,260 out (3,207 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 187 in → 2,339 out (2,000 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 185 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 184 in → 78 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 184 in → 2,681 out (2,355 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-31b-it:low PASS 100% 186 in → 2,660 out (2,304 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
ibm-granite/granite-4.2-8b:none__json_schema PASS 100% 163 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inception/mercury-2.5:high__json_schema PASS 100% 369 in → 1,733 out (1,607 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inception/mercury-2.5:low__json_schema PASS 100% 369 in → 386 out (250 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inception/mercury-2.5:max PASS 100% 154 in → 1,201 out (1,148 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
inception/mercury-2.5:max__json_schema PASS 100% 374 in → 1,765 out (1,634 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inception/mercury-2.5:medium PASS 100% 155 in → 1,064 out (1,013 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
inception/mercury-2.5:none PASS 100% 154 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
inception/mercury-2.5:none__json_schema PASS 100% 374 in → 158 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inception/mercury-2.5:xhigh__json_schema PASS 100% 376 in → 1,772 out (1,626 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
inclusionai/ling-2.6-flash__json_schema PASS 100% 178 in → 52 out
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
meta/muse-spark-1.3-contributor:low PASS 100% 138 in → 1,020 out (926 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
meta/muse-spark-1.3-contributor:xhigh PASS 100% 146 in → 7,220 out (7,090 reasoning)
git worktree add -b feature ../feature-wt || git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
meta/muse-spark-1.3-contributor:xhigh__json_schema PASS 100% 144 in → 6,474 out (6,210 reasoning)
git worktree add ../feature-wt feature || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature || git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
minimax/minimax-m2.7:high PASS 100% 171 in → 1,451 out (1,483 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
minimax/minimax-m2.7:low PASS 100% 173 in → 978 out (928 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
minimax/minimax-m2.7:low__json_schema PASS 100% 323 in → 924 out (855 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
minimax/minimax-m2.7:medium__json_schema PASS 100% 318 in → 857 out (789 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
minimax/minimax-m2.7:xhigh PASS 100% 174 in → 1,376 out (1,324 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 177 in → 722 out (688 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
mistralai/devstral-2512__json_schema PASS 100% 165 in → 67 out
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 172 in → 2,090 out (1,953 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 173 in → 2,194 out (2,138 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
moonshotai/kimi-k3:medium PASS 100% 215 in → 394 out (330 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
moonshotai/kimi-k3:xhigh PASS 100% 214 in → 1,577 out (1,513 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 220 in → 197 out (133 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 183 in → 678 out (592 reasoning)
git worktree add ../feature-wt feature echo new file > '../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 179 in → 819 out (795 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 176 in → 853 out (855 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 182 in → 357 out (326 reasoning)
git worktree add ../feature-wt feature printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 177 in → 394 out (349 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 184 in → 71 out (0 reasoning)
git worktree add '../feature-wt' HEAD echo 'new file' > '../feature-wt/new.txt' git -C '../feature-wt' add new.txt git -C '../feature-wt' commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add '../feature-wt' HEAD", "echo 'new file' > '../feature-wt/new.txt'", "git -C '../feature-wt' add new.txt", "git -C '../feature-wt' commit -m 'Add new file'" ] }
openai/gpt-5.4-nano:high PASS 100% 145 in → 549 out (450 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_clc2pbsz/worktree_usage_f008 git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:high__json_schema PASS 100% 185 in → 221 out (155 reasoning)
git worktree add ../feature-wt feature printf %s "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf %s \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
openai/gpt-5.4-nano:low PASS 100% 143 in → 175 out (120 reasoning)
git worktree add ../feature-wt feature printf "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:low__json_schema PASS 100% 192 in → 110 out (0 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qrfgp1bi/worktree_usage_f008 git worktree add ../feature-wt feature printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qrfgp1bi/worktree_usage_f008","git worktree add ../feature-wt feature","printf '%s' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:medium PASS 100% 142 in → 199 out (144 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:medium__json_schema PASS 100% 184 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:none PASS 100% 138 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature printf "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:none__json_schema PASS 100% 181 in → 109 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_b3yzxpeg/worktree_usage_f008 worktree add ../feature-wt feature printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_b3yzxpeg/worktree_usage_f008 worktree add ../feature-wt feature","printf '%s' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:xhigh PASS 100% 140 in → 2,009 out (1,954 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 186 in → 674 out (610 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
openai/gpt-5.5:high__json_schema PASS 100% 187 in → 580 out (516 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.5:medium PASS 100% 143 in → 335 out (233 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0h1f3ii1/worktree_usage_f008 git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-5.5:none__json_schema PASS 100% 185 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.6-luna:high__json_schema PASS 100% 187 in → 306 out (242 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.6-sol:none__json_schema PASS 100% 186 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-6-astra:max PASS 100% 144 in → 2,014 out (1,955 reasoning)
git worktree add ../feature-wt feature printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-6-astra:max__json_schema PASS 100% 189 in → 2,686 out (2,588 reasoning)
git show-ref --verify --quiet refs/heads/feature || git branch feature git worktree add ../feature-wt feature printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git show-ref --verify --quiet refs/heads/feature || git branch feature", "git worktree add ../feature-wt feature", "printf '%s' 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
openai/gpt-6-astra:xhigh PASS 100% 141 in → 974 out (886 reasoning)
if git show-ref --verify --quiet refs/heads/feature; then git worktree add ../feature-wt feature; else git worktree add -b feature ../feature-wt; fi printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-6-astra:xhigh__json_schema PASS 100% 193 in → 1,143 out (1,034 reasoning)
if git show-ref --verify --quiet refs/heads/feature; then git worktree add ../feature-wt feature; else git worktree add -b feature ../feature-wt; fi printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "if git show-ref --verify --quiet refs/heads/feature; then git worktree add ../feature-wt feature; else git worktree add -b feature ../feature-wt; fi", "printf '%s' 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
openai/gpt-6-luna:low__json_schema PASS 100% 188 in → 167 out (103 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-m.1:high__json_schema PASS 100% 170 in → 1,712 out (1,655 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-m.1:xhigh PASS 100% 171 in → 1,172 out (1,121 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-m.1:xhigh__json_schema PASS 100% 164 in → 1,230 out (1,155 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
poolside/laguna-xs-2.1:low PASS 100% 167 in → 835 out (785 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 166 in → 472 out (413 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-xs-2.1:none PASS 100% 171 in → 88 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file' </think>git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
poolside/laguna-xs-2.1:none__json_schema PASS 100% 166 in → 74 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
poolside/laguna-xs.2:high PASS 100% 205 in → 688 out (636 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-xs.2:low__json_schema PASS 100% 204 in → 259 out (200 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
poolside/laguna-xs.2:medium PASS 100% 204 in → 1,058 out (1,004 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-xs.2:none__json_schema PASS 100% 203 in → 58 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'"]}
poolside/laguna-xs.2:xhigh PASS 100% 207 in → 289 out (238 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
prism-ml/ternary-bonsai-2-27b:low__json_schema PASS 100% 270 in → 527 out (471 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
prism-ml/ternary-bonsai-2-27b:max PASS 100% 212 in → 1,694 out (1,642 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
prism-ml/ternary-bonsai-2-27b:medium PASS 100% 170 in → 396 out (344 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
prism-ml/ternary-bonsai-2-27b:medium__json_schema PASS 100% 274 in → 325 out (269 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
prism-ml/ternary-bonsai-2-27b:none__json_schema PASS 100% 277 in → 71 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file' cd ..
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'", "cd .." ] }
prism-ml/ternary-bonsai-2-27b:xhigh PASS 100% 214 in → 507 out (452 reasoning)
git worktree add ../feature-wt feature echo 'new file' > '../feature-wt/new.txt' git -C '../feature-wt' add new.txt git -C '../feature-wt' commit -m 'Add new file'
prism-ml/ternary-bonsai-2-27b:xhigh__json_schema PASS 100% 312 in → 2,517 out (2,441 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
qwen/qwen3.6-27b:high__json_schema PASS 100% 171 in → 2,222 out (1,946 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
qwen/qwen3.6-27b:medium PASS 100% 170 in → 1,935 out (1,701 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-27b:none PASS 100% 167 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-27b:none__json_schema PASS 100% 168 in → 74 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 167 in → 2,637 out (2,309 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 169 in → 2,449 out (2,151 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
qwen/qwen3.6-35b-a3b:none PASS 100% 172 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 173 in → 123 out (0 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_37s173jh/worktree_usage_f008 && git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_37s173jh/worktree_usage_f008 && git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 167 in → 2,663 out (2,457 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 170 in → 2,128 out (1,896 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
qwen/qwen3.6-flash:low PASS 100% 166 in → 2,591 out (2,537 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-flash:none PASS 100% 172 in → 59 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt checkout feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-flash:xhigh PASS 100% 172 in → 2,183 out (2,129 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-flash:xhigh PASS 100% 170 in → 1,765 out (1,711 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-max:none PASS 100% 173 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-max:xhigh PASS 100% 166 in → 1,937 out (1,883 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
qwen/qwen3.7-plus:high PASS 100% 168 in → 404 out (350 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-plus:medium PASS 100% 171 in → 2,385 out (2,331 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.8-flash:high__json_schema PASS 100% 223 in → 1,447 out (1,335 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_szr36jjt/worktree_usage_f008 worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_szr36jjt/worktree_usage_f008 worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
qwen/qwen3.8-flash:max__json_schema PASS 100% 226 in → 3,569 out (3,519 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
qwen/qwen3.8-flash:medium__json_schema PASS 100% 221 in → 2,125 out (2,009 reasoning)
git worktree add '../feature-wt' 'feature' echo 'new file' > '../feature-wt/new.txt' cd '../feature-wt' && git add 'new.txt' && git commit -m 'Add new file' cd '/private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_u64vm_v1/worktree_usage_f008'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add '../feature-wt' 'feature'", "echo 'new file' > '../feature-wt/new.txt'", "cd '../feature-wt' && git add 'new.txt' && git commit -m 'Add new file'", "cd '/private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_u64vm_v1/worktree_usage_f008'"]}
qwen/qwen3.8-flash:xhigh PASS 100% 215 in → 643 out (591 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.8-max:low__json_schema PASS 100% 194 in → 311 out (250 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
sakana/fugu-max:low__json_schema PASS 100% 332 in → 13,634 out (13,578 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
sakana/fugu-max:medium PASS 100% 251 in → 25,044 out (24,974 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
sakana/fugu-max:xhigh PASS 100% 251 in → 37,264 out (37,210 reasoning)
git worktree add -B feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
stealth/ox-alpha:low PASS 100% 232 in → 51 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
tencent/hy3:none__json_schema PASS 100% 151 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:high__json_schema PASS 100% 154 in → 669 out (596 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
thinkingmachines/inkling-small:low PASS 100% 150 in → 117 out (62 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:low__json_schema PASS 100% 146 in → 77 out (5 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:max PASS 100% 150 in → 466 out (411 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:max__json_schema PASS 100% 150 in → 423 out (351 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:medium PASS 100% 148 in → 476 out (409 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:medium__json_schema PASS 100% 147 in → 353 out (289 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
thinkingmachines/inkling-small:none PASS 100% 144 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 151 in → 602 out (531 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:high PASS 100% 150 in → 452 out (397 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
thinkingmachines/inkling:low PASS 100% 153 in → 111 out (56 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:low__json_schema PASS 100% 150 in → 138 out (67 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:max PASS 100% 154 in → 1,927 out (1,872 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:medium PASS 100% 153 in → 178 out (122 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:none__json_schema PASS 100% 155 in → 63 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:xhigh__json_schema PASS 100% 149 in → 400 out (329 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:high__json_schema PASS 100% 382 in → 1,786 out (1,720 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:low PASS 100% 321 in → 588 out (539 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:low__json_schema PASS 100% 386 in → 748 out (695 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:max PASS 100% 315 in → 636 out (587 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:max__json_schema PASS 100% 387 in → 1,784 out (1,718 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:medium PASS 100% 325 in → 1,135 out (1,086 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:medium__json_schema PASS 100% 390 in → 1,161 out (1,108 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:none PASS 100% 313 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:none__json_schema PASS 100% 383 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:xhigh__json_schema PASS 100% 382 in → 1,795 out (1,729 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
xiaomi/mimo-v2.6-flash:low__json_schema PASS 100% 168 in → 85 out (16 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
xiaomi/mimo-v2.6-flash:max__json_schema PASS 100% 164 in → 74 out (12 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
xiaomi/mimo-v2.6-flash:none PASS 100% 171 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
xiaomi/mimo-v2.6-flash:none__json_schema PASS 100% 169 in → 61 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
xiaomi/mimo-v2.6-flash:xhigh PASS 100% 165 in → 68 out (13 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
xiaomi/mimo-v2.6-flash:xhigh__json_schema PASS 100% 164 in → 71 out (21 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
xiaomi/mimo-v2.6-pro:medium PASS 100% 166 in → 68 out (17 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
xiaomi/mimo-v2.6-pro:medium__json_schema PASS 100% 168 in → 364 out (15 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git worktree add ../feature-wt feature cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && printf 'new file' > ../feature-wt/new.txt cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git -C ../feature-wt add new.txt cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008", "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git worktree add ../feature-wt feature", "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && printf 'new file' > ../feature-wt/new.txt", "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git -C ../feature-wt add new.txt", "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_xkag2l1a/worktree_usage_f008 && git -C ../feature-wt commit -m 'Add new file'"] }
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 148 in → 1,136 out (1,014 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
z-ai/glm-4.7-flash:none__json_schema PASS 100% 146 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
z-ai/glm-5.2:low PASS 100% 145 in → 2,728 out (2,495 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
z-ai/glm-5.2:none PASS 100% 147 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
z-ai/glm-5.3-flash:max PASS 100% 153 in → 1,045 out (994 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-5:high FAIL 50% 197 in → 96 out (22 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:low FAIL 50% 193 in → 80 out (0 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:low__json_schema FAIL 50% 433 in → 86 out (0 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:medium FAIL 50% 193 in → 119 out (31 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:none FAIL 50% 193 in → 80 out (0 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:none__json_schema FAIL 50% 427 in → 85 out (0 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:xhigh FAIL 50% 192 in → 115 out (27 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 50% 430 in → 340 out (74 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
arcee-ai/trinity-mini:high FAIL 50% 144 in → 624 out (606 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:high__json_schema FAIL 50% 140 in → 1,646 out (1,782 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:low__json_schema FAIL 50% 143 in → 663 out (609 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:medium FAIL 50% 146 in → 1,414 out (1,443 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:high FAIL 50% 181 in → 1,775 out (1,523 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:xhigh FAIL 50% 184 in → 2,650 out (2,424 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:medium FAIL 50% 186 in → 467 out (354 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:none FAIL 50% 185 in → 47 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:none__json_schema FAIL 50% 182 in → 56 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","cd ../feature-wt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:xhigh__json_schema FAIL 50% 188 in → 1,063 out (842 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:medium__json_schema FAIL 50% 170 in → 69 out (0 reasoning)
git worktree add ../feature-wt echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:high FAIL 50% 171 in → 1,139 out (1,126 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/devstral-2512 FAIL 50% 167 in → 48 out
```bash git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:none FAIL 50% 176 in → 48 out (0 reasoning)
```sh git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 50% 179 in → 53 out (0 reasoning)
git worktree add '../feature-wt' HEAD touch '../feature-wt/new.txt' echo 'new file' > '../feature-wt/new.txt' cd '../feature-wt' git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:none FAIL 50% 185 in → 44 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:low FAIL 50% 205 in → 75 out (22 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:high FAIL 50% 170 in → 1,101 out (1,055 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:medium FAIL 50% 172 in → 668 out (624 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 50% 164 in → 736 out (683 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:high__json_schema FAIL 50% 205 in → 203 out (136 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:none FAIL 50% 210 in → 45 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
prism-ml/ternary-bonsai-2-27b:none FAIL 50% 174 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file' cd ../feature-wt/..
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:low FAIL 50% 173 in → 2,044 out (1,864 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:xhigh__json_schema FAIL 50% 173 in → 2,588 out (1 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:low FAIL 50% 170 in → 2,909 out (2,861 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 50% 172 in → 2,043 out (1,823 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-flash:high FAIL 50% 167 in → 1,668 out (1,620 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-flash:medium FAIL 50% 166 in → 1,725 out (1,677 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-flash:none FAIL 50% 169 in → 43 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-plus:none FAIL 50% 175 in → 43 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-flash:xhigh__json_schema FAIL 50% 222 in → 682 out (629 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
stealth/ox-alpha:max FAIL 50% 225 in → 319 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
thinkingmachines/inkling:none FAIL 50% 153 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:high FAIL 50% 166 in → 69 out (20 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:high__json_schema FAIL 50% 168 in → 148 out (84 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:low FAIL 50% 147 in → 1,062 out (992 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:xhigh FAIL 50% 149 in → 2,051 out (2,022 reasoning)
git worktree add ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:medium FAIL 50% 154 in → 49 out (0 reasoning)
```bash git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5.1:high__json_schema FAIL 0% 434 in → 90 out (0 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:high FAIL 0% 193 in → 109 out (31 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:low FAIL 0% 198 in → 81 out (0 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:max FAIL 0% 201 in → 683 out (154 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:medium FAIL 0% 192 in → 104 out (22 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:xhigh FAIL 0% 197 in → 520 out (185 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-haiku-4.5:xhigh FAIL 0% 197 in → 9,852 out (3,397 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 350 in → 120 out (31 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:max FAIL 0% 169 in → 378 out (176 reasoning)
``` git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 352 in → 798 out (293 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:medium FAIL 0% 172 in → 67 out (0 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 348 in → 120 out (31 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.7:max FAIL 0% 200 in → 389 out (62 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:high FAIL 0% 201 in → 91 out (10 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:low FAIL 0% 193 in → 81 out (0 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:max FAIL 0% 195 in → 531 out (40 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 428 in → 896 out (89 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:medium FAIL 0% 194 in → 87 out (15 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:none FAIL 0% 195 in → 79 out (0 reasoning)
``` git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:xhigh FAIL 0% 198 in → 77 out (0 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-5:high FAIL 0% 198 in → 272 out (0 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-5:max FAIL 0% 196 in → 524 out (66 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-5:medium FAIL 0% 190 in → 320 out (7 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-5:xhigh FAIL 0% 197 in → 409 out (72 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:high FAIL 0% 172 in → 375 out (177 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 350 in → 328 out (179 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:max FAIL 0% 171 in → 581 out (205 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 352 in → 817 out (208 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 166 in → 425 out (156 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 352 in → 390 out (194 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-large-thinking:high FAIL 0% 142 in → 1,129 out (1,090 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low FAIL 0% 139 in → 2,272 out (2,237 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium FAIL 0% 140 in → 1,195 out (1,168 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low FAIL 0% 140 in → 1,012 out (1,025 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 147 in → 825 out (791 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 139 in → 759 out (726 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 142 in → 621 out (566 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:high FAIL 0% 146 in → 477 out (413 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 139 in → 883 out (757 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:low FAIL 0% 225 in → 2,678 out (2,626 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 142 in → 598 out (513 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:max FAIL 0% 233 in → 14,603 out (14,551 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:max__json_schema FAIL 0% 224 in → 596 out (524 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file\n' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 151 in → 364 out (425 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 142 in → 674 out (617 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:none FAIL 0% 142 in → 42 out (0 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 144 in → 62 out (0 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:xhigh FAIL 0% 230 in → 263 out (206 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 144 in → 316 out (252 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:high FAIL 0% 144 in → 600 out (557 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 251 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' cd -
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","cd ../feature-wt","echo 'new file' > new.txt","git add new.txt","git commit -m 'Add new file'","cd -"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 144 in → 699 out (623 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:medium FAIL 0% 143 in → 664 out (621 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:none FAIL 0% 147 in → 44 out (0 reasoning)
```bash git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:xhigh FAIL 0% 222 in → 1,555 out (1,502 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 224 in → 1,149 out (1,106 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:high FAIL 0% 142 in → 793 out (751 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:low FAIL 0% 143 in → 603 out (550 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 148 in → 528 out (416 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_uifjpqq8/worktree_usage_f008 git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_uifjpqq8/worktree_usage_f008", "git worktree add ../feature-wt -b feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:medium FAIL 0% 144 in → 879 out (801 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:none FAIL 0% 142 in → 40 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 142 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:xhigh FAIL 0% 143 in → 1,072 out (1,019 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 418 in → 4,230 out (4,168 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:high FAIL 0% 168 in → 2,029 out (1,976 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:high__json_schema FAIL 0% 172 in → 524 out (462 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:low FAIL 0% 168 in → 1,275 out (1,227 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:low__json_schema FAIL 0% 173 in → 1,456 out (1,408 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt","echo 'new file' > new.txt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:medium FAIL 0% 168 in → 1,160 out (1,099 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:medium__json_schema FAIL 0% 167 in → 1,121 out (1,058 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:none FAIL 0% 146 in → 40 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:none__json_schema FAIL 0% 148 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature -b feature git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": ["git worktree add ../feature-wt feature -b feature", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:xhigh FAIL 0% 175 in → 2,613 out (2,560 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4.1-flash:xhigh__json_schema FAIL 0% 168 in → 2,083 out (2,024 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:high FAIL 0% 167 in → 1,296 out (1,242 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:high__json_schema FAIL 0% 177 in → 1,133 out (1,075 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:low FAIL 0% 174 in → 881 out (827 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:medium FAIL 0% 169 in → 1,544 out (1,490 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 171 in → 750 out (692 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:xhigh FAIL 0% 167 in → 1,854 out (1,800 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 251 in → 683 out (625 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:high FAIL 0% 171 in → 1,268 out (1,214 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 248 in → 1,174 out (1,112 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:xhigh FAIL 0% 173 in → 1,944 out (1,890 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 250 in → 2,262 out (2,204 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:low FAIL 0% 172 in → 1,504 out (1,450 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 250 in → 843 out (785 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:medium FAIL 0% 171 in → 3,035 out (2,981 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:xhigh FAIL 0% 168 in → 2,914 out (2,860 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:high FAIL 0% 171 in → 1,433 out (1,379 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:xhigh FAIL 0% 175 in → 1,780 out (1,726 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 168 in → 1,115 out (1,081 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:high FAIL 0% 170 in → 2,196 out (2,153 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:high__json_schema FAIL 0% 241 in → 1,847 out (1,769 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:low FAIL 0% 170 in → 941 out (898 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:low__json_schema FAIL 0% 245 in → 593 out (515 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:medium FAIL 0% 169 in → 2,051 out (2,008 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:medium__json_schema FAIL 0% 247 in → 1,111 out (1,032 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:xhigh FAIL 0% 170 in → 1,258 out (1,204 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:xhigh__json_schema FAIL 0% 246 in → 1,236 out (1,157 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:high FAIL 0% 174 in → 1,994 out (1,940 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:high__json_schema FAIL 0% 174 in → 3,144 out (3,111 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:low FAIL 0% 175 in → 743 out (689 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:low__json_schema FAIL 0% 171 in → 55 out (0 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:max__json_schema FAIL 0% 171 in → 1,697 out (1,663 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:medium FAIL 0% 172 in → 2,342 out (2,288 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:medium__json_schema FAIL 0% 244 in → 706 out (648 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:xhigh FAIL 0% 174 in → 2,830 out (2,776 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:xhigh__json_schema FAIL 0% 172 in → 1,284 out (1,250 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.8-flash:high FAIL 0% 106 in → 4,070 out (0 reasoning)
(empty output)
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.8-flash:low FAIL 0% 172 in → 54 out (0 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.8-flash:low__json_schema FAIL 0% 246 in → 79 out (0 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 186 in → 3,118 out (2,614 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:low FAIL 0% 185 in → 4,419 out (3,748 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0% 183 in → 2,120 out (1,724 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
Failure: Expecting value: line 255 column 1 (char 1397)
google/gemma-4-31b-it:high__json_schema FAIL 0% 187 in → 1,742 out (1,465 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:low__json_schema FAIL 0% 179 in → 1,842 out (1,563 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:medium__json_schema FAIL 0% 186 in → 2,187 out (2,127 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:xhigh FAIL 0% 188 in → 2,200 out (1,863 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.1-8b FAIL 0% 158 in → 54 out
```bash git worktree add ../feature-wt main echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
git worktree add ../feature-wt main echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt main", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:high FAIL 0% 165 in → 3,725 out (3,522 reasoning)
git worktree add ../feature-wt cd ../feature-wt git checkout -b feature echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:high__json_schema FAIL 0% 162 in → 79 out (0 reasoning)
git worktree add ../feature-wt git checkout ../feature-wt echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt", "git checkout ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:low FAIL 0% 173 in → 64 out (7 reasoning)
worktree add ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add ../feature-wt/new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:low__json_schema FAIL 0% 176 in → 68 out (0 reasoning)
git worktree add ../feature-wt git checkout ../feature-wt echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file' -f
JSON Schema Structured Output
(raw) { "commands": ["git worktree add ../feature-wt", "git checkout ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file' -f"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:max FAIL 0% 167 in → 2,149 out (2,021 reasoning)
git worktree add ../feature-wt --branch feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:max__json_schema FAIL 0% 164 in → 68 out (0 reasoning)
git worktree add ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:medium FAIL 0% 176 in → 55 out (8 reasoning)
git worktree add ../feature-wt cd ../feature-wt echo "new file" > new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:none FAIL 0% 164 in → 40 out (0 reasoning)
worktree create ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:xhigh FAIL 0% 164 in → 2,746 out (2,617 reasoning)
git branch feature git worktree add ../feature-wt feature cd ../feature-wt printf 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.2-8b:xhigh__json_schema FAIL 0% 169 in → 84 out (0 reasoning)
git worktree add ../feature-wt git checkout -b feature ../feature-wt/HEAD echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt", "git checkout -b feature ../feature-wt/HEAD", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inception/mercury-2.5:high FAIL 0% 153 in → 942 out (901 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inception/mercury-2.5:low FAIL 0% 152 in → 288 out (246 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inception/mercury-2.5:medium__json_schema FAIL 0% 382 in → 1,508 out (1,390 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inception/mercury-2.5:xhigh FAIL 0% 152 in → 980 out (939 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inclusionai/ling-2.6-flash FAIL 0% 177 in → 40 out
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
liquid/lfm-2-24b-a2b FAIL 0% 162 in → 43 out
```bash cd ../feature-wt git worktree add main feature echo "new file" > new.txt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: ``` cd ../feature-wt git worktree add ../feature feature echo "new file" > new.txt git add new.txt git commit -m "Add new file" ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
meta/muse-spark-1.1:high FAIL 0% 141 in → 1,868 out (1,805 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:high__json_schema FAIL 0% 139 in → 1,566 out (1,488 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:low FAIL 0% 142 in → 827 out (766 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:low__json_schema FAIL 0% 140 in → 1,065 out (994 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:max FAIL 0% 142 in → 2,198 out (2,137 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:max__json_schema FAIL 0% 141 in → 1,712 out (1,641 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:medium FAIL 0% 140 in → 2,041 out (1,980 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:medium__json_schema FAIL 0% 141 in → 1,557 out (1,486 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:xhigh FAIL 0% 142 in → 1,838 out (1,777 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.1:xhigh__json_schema FAIL 0% 142 in → 2,355 out (2,283 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.3-contributor:high FAIL 0% 144 in → 3,546 out (3,382 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.3-contributor:high__json_schema FAIL 0% 144 in → 3,217 out (3,045 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.3-contributor:low__json_schema FAIL 0% 148 in → 1,583 out (1,451 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.3-contributor:medium FAIL 0% 142 in → 2,570 out (2,432 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
meta/muse-spark-1.3-contributor:medium__json_schema FAIL 0% 140 in → 2,092 out (1,955 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:high__json_schema FAIL 0% 174 in → 684 out (602 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:low FAIL 0% 169 in → 2,020 out (1,954 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:low__json_schema FAIL 0% 187 in → 989 out (928 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:medium FAIL 0% 173 in → 1,456 out (1,392 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:medium__json_schema FAIL 0% 177 in → 928 out (875 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:xhigh FAIL 0% 166 in → 1,874 out (1,827 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 173 in → 1,910 out (1,791 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.7:high__json_schema FAIL 0% 323 in → 1,592 out (1,521 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.7:medium FAIL 0% 173 in → 1,228 out (1,156 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:high FAIL 0% 307 in → 1,130 out (1,085 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:high__json_schema FAIL 0% 296 in → 70 out (0 reasoning)
git worktree add -b feature ../feature-wt HEAD echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt HEAD", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:low FAIL 0% 310 in → 1,409 out (1,336 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:low__json_schema FAIL 0% 261 in → 59 out (58 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt && git switch feature && echo 'new file' > new.txt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "cd ../feature-wt && git switch feature && echo 'new file' > new.txt && git add new.txt && git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:medium FAIL 0% 315 in → 518 out (491 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:medium__json_schema FAIL 0% 299 in → 79 out (0 reasoning)
git worktree add -b feature ../feature-wt touch ../feature-wt/new.txt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "touch ../feature-wt/new.txt", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:xhigh FAIL 0% 309 in → 735 out (647 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:xhigh__json_schema FAIL 0% 257 in → 100 out (99 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mmjbsbu9/worktree_usage_f008 git worktree add -b feature ../feature-wt cd ../feature-wt touch new.txt printf 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mmjbsbu9/worktree_usage_f008", "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "touch new.txt", "printf 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:high FAIL 0% 181 in → 3,552 out (3,800 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:low FAIL 0% 179 in → 3,616 out (3,838 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:medium FAIL 0% 181 in → 3,269 out (3,345 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 167 in → 2,192 out (2,279 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 164 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:xhigh FAIL 0% 181 in → 2,641 out (2,708 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 166 in → 2,690 out (2,714 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:high FAIL 0% 140 in → 1,928 out (1,875 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 135 in → 2,787 out (2,812 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:low FAIL 0% 139 in → 3,111 out (3,110 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 142 in → 3,369 out (2,810 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:medium FAIL 0% 143 in → 3,189 out (3,150 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.6:xhigh FAIL 0% 137 in → 1,627 out (1,598 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 140 in → 4,933 out (4,870 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:high FAIL 0% 143 in → 2,911 out (2,858 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:low FAIL 0% 140 in → 528 out (481 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 142 in → 387 out (320 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:medium FAIL 0% 136 in → 798 out (793 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0% 138 in → 348 out (298 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 137 in → 1,106 out (1,121 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0% 141 in → 680 out (638 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:high FAIL 0% 214 in → 899 out (833 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:high__json_schema FAIL 0% 215 in → 784 out (718 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:low FAIL 0% 223 in → 785 out (719 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:low__json_schema FAIL 0% 215 in → 1,027 out (961 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:max FAIL 0% 217 in → 1,063 out (997 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:max__json_schema FAIL 0% 216 in → 639 out (573 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:medium__json_schema FAIL 0% 216 in → 1,121 out (1,055 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 181 in → 500 out (454 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 179 in → 459 out (439 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 185 in → 600 out (576 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:high FAIL 0% 182 in → 390 out (347 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 182 in → 278 out (218 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 184 in → 335 out (289 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 181 in → 262 out (197 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:medium FAIL 0% 180 in → 497 out (468 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 181 in → 402 out (331 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 180 in → 59 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 0% 181 in → 206 out (162 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 180 in → 417 out (356 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:high FAIL 0% 147 in → 691 out (516 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/worktree_usage_f008 && git worktree add -b feature ../feature-wt printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/feature-wt/new.txt cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/feature-wt && git add new.txt && git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:high__json_schema FAIL 0% 188 in → 582 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:low FAIL 0% 141 in → 115 out (58 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:low__json_schema FAIL 0% 184 in → 148 out (80 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt && printf 'new file' > new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt && printf 'new file' > new.txt","cd ../feature-wt && git add new.txt","cd ../feature-wt && git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:medium FAIL 0% 142 in → 468 out (411 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:medium__json_schema FAIL 0% 188 in → 311 out (245 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:none FAIL 0% 145 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:none__json_schema FAIL 0% 186 in → 219 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/worktree_usage_f008 worktree add ../feature-wt -b feature printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/worktree_usage_f008 worktree add ../feature-wt -b feature","printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt/new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt add new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:xhigh FAIL 0% 138 in → 27,516 out (27,282 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/worktree_usage_f008 worktree add -b feature ../feature-wt d4051ae41985470992f5db41630baf3c29b32123 printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt add -- new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:xhigh__json_schema FAIL 0% 186 in → 25,124 out (24,862 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/worktree_usage_f008 worktree add -b feature /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt HEAD printf '%s' 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt/new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt add new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/worktree_usage_f008 worktree add -b feature /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt HEAD","printf '%s' 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt/new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt add new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:high FAIL 0% 141 in → 619 out (516 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_zlckm3pc/worktree_usage_f008 git worktree add -b feature ../feature-wt HEAD printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:high__json_schema FAIL 0% 187 in → 250 out (183 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:low FAIL 0% 141 in → 285 out (68 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:low__json_schema FAIL 0% 191 in → 128 out (61 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:medium FAIL 0% 142 in → 461 out (360 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qb991ew1/worktree_usage_f008 git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:medium__json_schema FAIL 0% 191 in → 204 out (137 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:none FAIL 0% 140 in → 182 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_n4j84znx/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:none__json_schema FAIL 0% 189 in → 65 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:xhigh FAIL 0% 141 in → 275 out (217 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:xhigh__json_schema FAIL 0% 185 in → 9,411 out (9,345 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:high FAIL 0% 142 in → 1,655 out (1,552 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mq4g8ke6/worktree_usage_f008 git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:low FAIL 0% 146 in → 208 out (150 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:low__json_schema FAIL 0% 189 in → 139 out (73 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:medium__json_schema FAIL 0% 187 in → 426 out (360 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:none FAIL 0% 147 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:xhigh FAIL 0% 141 in → 574 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:xhigh__json_schema FAIL 0% 186 in → 582 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:high FAIL 0% 139 in → 176 out (119 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:low FAIL 0% 141 in → 133 out (76 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:low__json_schema FAIL 0% 193 in → 183 out (117 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:max FAIL 0% 145 in → 305 out (248 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:max__json_schema FAIL 0% 189 in → 1,100 out (1,034 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:medium FAIL 0% 143 in → 204 out (147 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:medium__json_schema FAIL 0% 188 in → 204 out (138 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:none FAIL 0% 141 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:none__json_schema FAIL 0% 188 in → 64 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:xhigh FAIL 0% 143 in → 270 out (213 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:xhigh__json_schema FAIL 0% 185 in → 289 out (223 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:high FAIL 0% 140 in → 179 out (119 reasoning)
git worktree add -b feature ../feature-wt printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:high__json_schema FAIL 0% 190 in → 584 out (357 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/worktree_usage_f008 worktree add -b feature ../feature-wt","printf 'new file\\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt/new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt add new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:low FAIL 0% 137 in → 123 out (62 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:low__json_schema FAIL 0% 191 in → 140 out (69 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:max FAIL 0% 140 in → 573 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:max__json_schema FAIL 0% 188 in → 584 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:medium FAIL 0% 141 in → 102 out (41 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 188 in → 165 out (97 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:none FAIL 0% 138 in → 59 out (0 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:xhigh FAIL 0% 139 in → 188 out (131 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 185 in → 273 out (202 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:high FAIL 0% 146 in → 150 out (92 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:high__json_schema FAIL 0% 185 in → 162 out (94 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:low FAIL 0% 140 in → 115 out (57 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:low__json_schema FAIL 0% 189 in → 116 out (48 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:max FAIL 0% 137 in → 389 out (331 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:max__json_schema FAIL 0% 187 in → 282 out (211 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:medium FAIL 0% 141 in → 143 out (85 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 190 in → 212 out (141 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:none FAIL 0% 140 in → 56 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:none__json_schema FAIL 0% 187 in → 66 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:xhigh FAIL 0% 138 in → 459 out (398 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 184 in → 208 out (137 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:high FAIL 0% 141 in → 188 out (130 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:high__json_schema FAIL 0% 187 in → 270 out (199 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:low FAIL 0% 141 in → 59 out (0 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:low__json_schema FAIL 0% 192 in → 148 out (77 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:medium FAIL 0% 140 in → 134 out (73 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-astra:medium__json_schema FAIL 0% 182 in → 152 out (84 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:high FAIL 0% 139 in → 228 out (171 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:high__json_schema FAIL 0% 186 in → 204 out (138 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:low FAIL 0% 137 in → 142 out (85 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:max FAIL 0% 141 in → 244 out (187 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:max__json_schema FAIL 0% 188 in → 225 out (159 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:medium FAIL 0% 143 in → 226 out (169 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:medium__json_schema FAIL 0% 187 in → 161 out (95 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:xhigh FAIL 0% 142 in → 318 out (260 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-luna:xhigh__json_schema FAIL 0% 186 in → 294 out (228 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:high FAIL 0% 143 in → 229 out (172 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:high__json_schema FAIL 0% 189 in → 246 out (178 reasoning)
git worktree add -b feature ../feature-wt printf %s 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf %s 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:low FAIL 0% 137 in → 96 out (39 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:low__json_schema FAIL 0% 190 in → 152 out (86 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:max FAIL 0% 139 in → 892 out (835 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:max__json_schema FAIL 0% 188 in → 1,508 out (1,442 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:medium FAIL 0% 139 in → 198 out (141 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:medium__json_schema FAIL 0% 186 in → 246 out (180 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:xhigh FAIL 0% 140 in → 376 out (319 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-6-sol:xhigh__json_schema FAIL 0% 187 in → 383 out (317 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:high FAIL 0% 200 in → 1,523 out (1,565 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:high__json_schema FAIL 0% 201 in → 814 out (750 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:low FAIL 0% 198 in → 88 out (43 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:low__json_schema FAIL 0% 205 in → 205 out (156 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt printf 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt","printf 'new file' > new.txt","git add new.txt","git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:medium FAIL 0% 204 in → 330 out (271 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:medium__json_schema FAIL 0% 189 in → 415 out (331 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:xhigh FAIL 0% 185 in → 1,230 out (1,276 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 204 in → 1,484 out (1,507 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:high FAIL 0% 201 in → 1,467 out (1,513 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:high__json_schema FAIL 0% 190 in → 596 out (554 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt","echo \"new file\" > new.txt","git add new.txt","git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:low__json_schema FAIL 0% 200 in → 97 out (43 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","cd ../feature-wt","echo 'new file' > new.txt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:medium FAIL 0% 200 in → 582 out (543 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:medium__json_schema FAIL 0% 197 in → 343 out (281 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:xhigh FAIL 0% 202 in → 1,282 out (1,234 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 189 in → 549 out (485 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:low FAIL 0% 167 in → 1,089 out (1,049 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:low__json_schema FAIL 0% 169 in → 2,624 out (2,551 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' cd -
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'", "cd -" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:medium FAIL 0% 165 in → 1,208 out (1,166 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:medium__json_schema FAIL 0% 171 in → 2,367 out (2,300 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:none FAIL 0% 172 in → 63 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt checkout -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:none__json_schema FAIL 0% 172 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:high FAIL 0% 171 in → 1,728 out (1,688 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 173 in → 1,215 out (1,148 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 167 in → 772 out (723 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:xhigh FAIL 0% 165 in → 815 out (775 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:low FAIL 0% 204 in → 783 out (727 reasoning)
git branch feature git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:medium__json_schema FAIL 0% 204 in → 2,056 out (2,001 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 205 in → 411 out (349 reasoning)
git worktree add -b feature ../feature-wt main echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt main", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
prism-ml/ternary-bonsai-2-27b:high FAIL 0% 174 in → 503 out (455 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
prism-ml/ternary-bonsai-2-27b:high__json_schema FAIL 0% 271 in → 779 out (701 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > '../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file' > '../feature-wt/new.txt'", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
prism-ml/ternary-bonsai-2-27b:low FAIL 0% 168 in → 289 out (235 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
prism-ml/ternary-bonsai-2-27b:max__json_schema FAIL 0% 309 in → 951 out (873 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:high FAIL 0% 169 in → 2,452 out (2,181 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 173 in → 2,218 out (1,974 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:xhigh FAIL 0% 171 in → 3,700 out (3,650 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 164 in → 2,048 out (1,832 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:medium FAIL 0% 168 in → 3,358 out (2,946 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abe29d1b-c2ff-9b55-8b53-d704ddfa9d3a","request_id":"abe29d1b-c2ff-9b55-8b53-d704ddfa9d3a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6eeedb1-9704-921b-a1bb-95583a73de4e","request_id":"e6eeedb1-9704-921b-a1bb-95583a73de4e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium FAIL 0% 169 in → 1,913 out (1,857 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2e73abef-e4ce-9a98-9e87-5b220289f0e7","request_id":"2e73abef-e4ce-9a98-9e87-5b220289f0e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bf8874e1-8a6b-9276-aee2-1a03c1fa2e24","request_id":"bf8874e1-8a6b-9276-aee2-1a03c1fa2e24"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4fc7bbbb-0d7a-978b-aab3-81b681d29881","request_id":"4fc7bbbb-0d7a-978b-aab3-81b681d29881"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high FAIL 0% 168 in → 2,087 out (2,037 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4a27c070-d19e-922c-aa91-a5bd10610acc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low FAIL 0% 168 in → 2,276 out (2,227 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add . && git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dd9f1329-0efc-92b9-951f-9d25dd0d1502"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 171 in → 2,030 out (1,980 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5f5fd8f2-d7e4-93c4-9b25-f969fc670da0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9e5d9abe-23d7-9033-bd26-72a3025866a6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6621c924-107b-9c6a-adbf-e284575e66fb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-411db393-eb88-9a61-b39a-c604711b8f6e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high FAIL 0% 170 in → 1,188 out (1,132 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-06294a1b-49ee-9153-8d5c-aab2ed96261f","request_id":"06294a1b-49ee-9153-8d5c-aab2ed96261f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low FAIL 0% 174 in → 1,381 out (1,325 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90e71462-fd89-96a3-9d2a-1da6baac9533","request_id":"90e71462-fd89-96a3-9d2a-1da6baac9533"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium FAIL 0% 167 in → 1,735 out (1,679 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6c08186-7ca9-98c4-af7f-5cca12529b3e","request_id":"a6c08186-7ca9-98c4-af7f-5cca12529b3e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbde3990-7d19-957f-a366-ebd0bc041c9f","request_id":"fbde3990-7d19-957f-a366-ebd0bc041c9f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f3578958-71e2-9214-95ba-dd354e046d4a","request_id":"f3578958-71e2-9214-95ba-dd354e046d4a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ff7de57-3e0d-916e-9a88-7b88f31fced8","request_id":"2ff7de57-3e0d-916e-9a88-7b88f31fced8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low FAIL 0% 167 in → 2,111 out (2,055 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d22ee697-4e11-90b1-81e9-ce7978a4a3e6","request_id":"d22ee697-4e11-90b1-81e9-ce7978a4a3e6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4cca6aa0-c616-982d-ba32-0710b64b9c08","request_id":"4cca6aa0-c616-982d-ba32-0710b64b9c08"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a56e14e2-cabc-93ef-bf2b-39f8910c36e2","request_id":"a56e14e2-cabc-93ef-bf2b-39f8910c36e2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh FAIL 0% 171 in → 2,053 out (1,997 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-873573ec-17cf-9204-ae59-b4c4c729d79b","request_id":"873573ec-17cf-9204-ae59-b4c4c729d79b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.8-flash:high FAIL 0% 221 in → 1,730 out (1,676 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-flash:low FAIL 0% 221 in → 5,463 out (5,408 reasoning)
git worktree add -b feature ../feature-wt HEAD printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-flash:low__json_schema FAIL 0% 224 in → 13,017 out (12,903 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_prz9szvd/worktree_usage_f008 worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_prz9szvd/worktree_usage_f008 worktree add ../feature-wt -b feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-flash:max FAIL 0% 216 in → 1,134 out (1,079 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-flash:medium FAIL 0% 220 in → 5,586 out (5,531 reasoning)
git worktree add -b feature ../feature-wt main printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:high FAIL 0% 209 in → 4,291 out (4,235 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:high__json_schema FAIL 0% 208 in → 2,034 out (1,970 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:low FAIL 0% 200 in → 133 out (77 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:max FAIL 0% 207 in → 1,346 out (1,290 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:max__json_schema FAIL 0% 210 in → 1,383 out (1,258 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_og3srng5/worktree_usage_f008 git worktree add ../feature-wt -b feature cd ../feature-wt printf 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": [ "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_og3srng5/worktree_usage_f008", "git worktree add ../feature-wt -b feature", "cd ../feature-wt", "printf 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:medium FAIL 0% 170 in → 526 out (470 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:medium__json_schema FAIL 0% 176 in → 442 out (382 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:xhigh FAIL 0% 208 in → 7,889 out (7,833 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.8-max:xhigh__json_schema FAIL 0% 209 in → 6,700 out (6,570 reasoning)
Cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_j4cw0ala/worktree_usage_f008 Git worktree add ../feature-wt -b feature Cd ../feature-wt Echo 'new file' > new.txt Git add new.txt Git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "Cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_j4cw0ala/worktree_usage_f008", "Git worktree add ../feature-wt -b feature", "Cd ../feature-wt", "Echo 'new file' > new.txt", "Git add new.txt", "Git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:high FAIL 0% 215 in → 195 out (137 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:high__json_schema FAIL 0% 265 in → 929 out (702 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/worktree_usage_f008 && git worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/worktree_usage_f008 && git worktree add -b feature ../feature-wt","printf 'new file\\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt/new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt add new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_v4vehr9w/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:low FAIL 0% 218 in → 217 out (160 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:max FAIL 0% 218 in → 458 out (400 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:max__json_schema FAIL 0% 329 in → 33,448 out (33,390 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:medium__json_schema FAIL 0% 266 in → 258 out (192 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt && git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt && git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
sakana/fugu-max:xhigh__json_schema FAIL 0% 262 in → 564 out (493 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
stealth/ox-alpha:high FAIL 0% 224 in → 64 out (0 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt && echo 'new file' > new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: ```bash git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
tencent/hy3:high FAIL 0% 154 in → 5,439 out (5,386 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:high__json_schema FAIL 0% 152 in → 8,332 out (8,279 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:low FAIL 0% 149 in → 860 out (807 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:low__json_schema FAIL 0% 149 in → 1,427 out (1,276 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:medium FAIL 0% 152 in → 1,787 out (1,740 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:medium__json_schema FAIL 0% 153 in → 1,557 out (1,379 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file\\n' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:none FAIL 0% 155 in → 40 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:xhigh FAIL 0% 151 in → 11,354 out (11,301 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:xhigh__json_schema FAIL 0% 154 in → 3,584 out (3,531 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
thinkingmachines/inkling-small:high FAIL 0% 150 in → 671 out (630 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'{"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh FAIL 0% 153 in → 923 out (866 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh FAIL 0% 151 in → 648 out (590 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.3:high FAIL 0% 318 in → 4,613 out (4,562 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.3:xhigh FAIL 0% 317 in → 2,462 out (2,411 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:high FAIL 0% 352 in → 701 out (649 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:high__json_schema FAIL 0% 431 in → 1,019 out (962 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:low FAIL 0% 353 in → 371 out (316 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:low__json_schema FAIL 0% 430 in → 621 out (564 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:max FAIL 0% 355 in → 1,536 out (1,484 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:max__json_schema FAIL 0% 431 in → 1,476 out (1,403 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:medium FAIL 0% 350 in → 728 out (676 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:medium__json_schema FAIL 0% 428 in → 1,454 out (1,397 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:xhigh FAIL 0% 355 in → 1,107 out (1,055 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:xhigh__json_schema FAIL 0% 429 in → 1,499 out (1,426 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:high FAIL 0% 1,391 in → 842 out (788 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:high__json_schema FAIL 0% 1,472 in → 1,781 out (1,724 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:low FAIL 0% 1,388 in → 360 out (303 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:low__json_schema FAIL 0% 1,468 in → 499 out (441 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:max FAIL 0% 1,384 in → 1,042 out (1,028 reasoning)
I'll create the worktree, add the file, and commit it.
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:max__json_schema FAIL 0% 1,470 in → 2,384 out (2,327 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:medium FAIL 0% 1,386 in → 837 out (780 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:medium__json_schema FAIL 0% 1,460 in → 1,128 out (1,067 reasoning)
git worktree add -b feature ../feature-wt printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:xhigh FAIL 0% 1,391 in → 1,224 out (1,167 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.7:xhigh__json_schema FAIL 0% 1,465 in → 997 out (940 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:high__json_schema FAIL 0% 168 in → 88 out (23 reasoning)
git worktree add -b feature ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m \"Add new file\""] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:low FAIL 0% 166 in → 62 out (13 reasoning)
git worktree add -b feature ../feature-wt main echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:max FAIL 0% 166 in → 56 out (12 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:medium FAIL 0% 168 in → 131 out (83 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-flash:medium__json_schema FAIL 0% 165 in → 110 out (58 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","cd ../feature-wt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:high FAIL 0% 171 in → 114 out (61 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
Failure: OpenAI-compatible response contained provider error: Provider timed out after 10106ms
xiaomi/mimo-v2.6-pro:low__json_schema FAIL 0% 168 in → 193 out (128 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:max FAIL 0% 165 in → 97 out (44 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:max__json_schema FAIL 0% 164 in → 409 out (337 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file\\n' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:xhigh FAIL 0% 167 in → 102 out (49 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
xiaomi/mimo-v2.6-pro:xhigh__json_schema FAIL 0% 167 in → 326 out (57 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/worktree_usage_f008 worktree add ../feature-wt -b feature echo 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/worktree_usage_f008 worktree add ../feature-wt -b feature", "echo 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt/new.txt", "git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt add new.txt", "git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_jfj49old/feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:high FAIL 0% 143 in → 1,646 out (1,471 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 148 in → 1,228 out (1,201 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:none FAIL 0% 145 in → 39 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high FAIL 0% 144 in → 3,260 out (2,974 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:high__json_schema FAIL 0% 158 in → 992 out (981 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commands": [ ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:low__json_schema FAIL 0% 150 in → 788 out (724 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:medium FAIL 0% 155 in → 203 out (150 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:medium__json_schema FAIL 0% 338 in → 498 out (427 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:none__json_schema FAIL 0% 145 in → 66 out (0 reasoning)
git worktree add '../feature-wt' -b feature echo 'new file' > '../feature-wt/new.txt' git -C '../feature-wt' add new.txt git -C '../feature-wt' commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add '../feature-wt' -b feature", "echo 'new file' > '../feature-wt/new.txt'", "git -C '../feature-wt' add new.txt", "git -C '../feature-wt' commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:xhigh FAIL 0% 146 in → 567 out (438 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 157 in → 1,240 out (1,122 reasoning)
Cd to /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qivxl_d6/worktree_usage_f008 git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "Cd to /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qivxl_d6/worktree_usage_f008", "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:high FAIL 0% 152 in → 141 out (90 reasoning)
```bash git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:high__json_schema FAIL 0% 154 in → 280 out (223 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:low FAIL 0% 153 in → 64 out (10 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:low__json_schema FAIL 0% 158 in → 51 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","printf 'new file' > ../feature-wt/new.txt","cd ../feature-wt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:max__json_schema FAIL 0% 156 in → 2,198 out (2,135 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:medium__json_schema FAIL 0% 155 in → 352 out (290 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:xhigh FAIL 0% 149 in → 45 out (4 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.3-flash:xhigh__json_schema FAIL 0% 156 in → 184 out (127 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]