Create worktree and make commit
Tests ability to create a worktree and immediately make a commit in it. Evaluates end-to-end worktree commit workflow.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'Hello' > hello.txt
  5. 05 git add hello.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 git checkout -b feature
  8. 08 echo 'Feature' > feature.txt
  9. 09 git add feature.txt
  10. 10 git commit -m 'Add feature'
  11. 11 git checkout main
Prompt
Create a worktree for 'feature' at '../feature-wt', then create a file '../feature-wt/new.txt' with content 'new file', add it and commit with message 'Add new file' in the worktree. Output ONLY the commands to run, one per line.
Expected
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'

Scoped model quality, cost, API time, and token usage for worktree_usage/f008.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 429 in → 110 out (34 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:low__json_schema PASS 100% 431 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:max__json_schema PASS 100% 433 in → 825 out (101 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:medium__json_schema PASS 100% 433 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 432 in → 485 out (96 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:high PASS 100% 195 in → 412 out (277 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 381 in → 3,940 out (1,783 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:low PASS 100% 197 in → 735 out (549 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 384 in → 1,276 out (626 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:medium PASS 100% 198 in → 572 out (403 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 384 in → 1,009 out (646 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-haiku-4.5:none PASS 100% 168 in → 66 out (0 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 351 in → 86 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt status sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "git -C ../feature-wt status", "sh -c 'echo \"new file\" > ../feature-wt/new.txt'", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 384 in → 2,630 out (1,256 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:high PASS 100% 167 in → 120 out (37 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:low PASS 100% 173 in → 64 out (0 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 352 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 355 in → 99 out (15 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:none PASS 100% 168 in → 64 out (0 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 358 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.6:xhigh PASS 100% 172 in → 157 out (65 reasoning)
``` git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
anthropic/claude-opus-4.7:high PASS 100% 203 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
anthropic/claude-opus-4.7:high__json_schema PASS 100% 437 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:low PASS 100% 201 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:low__json_schema PASS 100% 431 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:max__json_schema PASS 100% 432 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:medium PASS 100% 202 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 436 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c "echo 'new file' > ../feature-wt/new.txt" git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c \"echo 'new file' > ../feature-wt/new.txt\"","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:none PASS 100% 203 in → 75 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
anthropic/claude-opus-4.7:none__json_schema PASS 100% 428 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c "echo 'new file' > ../feature-wt/new.txt" git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c \"echo 'new file' > ../feature-wt/new.txt\"","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.7:xhigh PASS 100% 202 in → 77 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 431 in → 94 out (0 reasoning)
git worktree add ../feature-wt feature sh -c 'echo "new file" > ../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","sh -c 'echo \"new file\" > ../feature-wt/new.txt'","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 428 in → 124 out (42 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:low__json_schema PASS 100% 433 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 431 in → 84 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 428 in → 83 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 435 in → 304 out (43 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:low PASS 100% 168 in → 104 out (9 reasoning)
git worktree add ../feature-wt feature 2>/dev/null || git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 351 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:medium PASS 100% 166 in → 150 out (60 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 355 in → 120 out (34 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-4.6:none PASS 100% 174 in → 66 out (0 reasoning)
``` git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 354 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-5:high__json_schema PASS 100% 433 in → 237 out (59 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 435 in → 166 out (38 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
arcee-ai/trinity-large-thinking:xhigh PASS 100% 142 in → 1,403 out (1,364 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
deepseek/deepseek-v4-flash:low PASS 100% 145 in → 530 out (485 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 251 in → 56 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 146 in → 68 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 148 in → 69 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 343 in → 585 out (528 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3-flash-preview:low__json_schema PASS 100% 247 in → 770 out (710 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.1-flash-lite-preview:low PASS 100% 172 in → 179 out (127 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 243 in → 170 out (114 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 171 in → 616 out (564 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 174 in → 376 out (364 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 169 in → 920 out (888 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-pro-preview:high PASS 100% 168 in → 2,569 out (2,517 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 249 in → 932 out (874 reasoning)
git worktree add -B feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -B feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 247 in → 4,048 out (3,992 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 248 in → 1,622 out (1,566 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.5-flash-lite:low PASS 100% 172 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 168 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
google/gemini-3.5-flash-lite:max PASS 100% 166 in → 1,953 out (1,901 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 243 in → 1,329 out (1,273 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.5-flash-lite:medium PASS 100% 170 in → 696 out (644 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 171 in → 451 out (419 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
google/gemini-3.6-flash:max PASS 100% 167 in → 1,802 out (1,750 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
google/gemma-4-26b-a4b-it:medium PASS 100% 191 in → 3,260 out (3,207 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 187 in → 2,339 out (2,000 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-26b-a4b-it:none PASS 100% 185 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 184 in → 78 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 184 in → 2,681 out (2,355 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
google/gemma-4-31b-it:low PASS 100% 186 in → 2,660 out (2,304 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
inclusionai/ling-2.6-flash__json_schema PASS 100% 178 in → 52 out
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
minimax/minimax-m2.7:high PASS 100% 171 in → 1,451 out (1,483 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
minimax/minimax-m2.7:low PASS 100% 173 in → 978 out (928 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
minimax/minimax-m2.7:low__json_schema PASS 100% 323 in → 924 out (855 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
minimax/minimax-m2.7:medium__json_schema PASS 100% 318 in → 857 out (789 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
minimax/minimax-m2.7:xhigh PASS 100% 174 in → 1,376 out (1,324 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 177 in → 722 out (688 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
mistralai/devstral-2512__json_schema PASS 100% 165 in → 67 out
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 172 in → 2,090 out (1,953 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 173 in → 2,194 out (2,138 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
moonshotai/kimi-k3:medium PASS 100% 215 in → 394 out (330 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
moonshotai/kimi-k3:xhigh PASS 100% 214 in → 1,577 out (1,513 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 220 in → 197 out (133 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 183 in → 678 out (592 reasoning)
git worktree add ../feature-wt feature echo new file > '../feature-wt/new.txt' git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 179 in → 819 out (795 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 176 in → 853 out (855 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 182 in → 357 out (326 reasoning)
git worktree add ../feature-wt feature printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 177 in → 394 out (349 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 184 in → 71 out (0 reasoning)
git worktree add '../feature-wt' HEAD echo 'new file' > '../feature-wt/new.txt' git -C '../feature-wt' add new.txt git -C '../feature-wt' commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add '../feature-wt' HEAD", "echo 'new file' > '../feature-wt/new.txt'", "git -C '../feature-wt' add new.txt", "git -C '../feature-wt' commit -m 'Add new file'" ] }
openai/gpt-5.4-nano:high PASS 100% 145 in → 549 out (450 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_clc2pbsz/worktree_usage_f008 git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:high__json_schema PASS 100% 185 in → 221 out (155 reasoning)
git worktree add ../feature-wt feature printf %s "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf %s \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
openai/gpt-5.4-nano:low PASS 100% 143 in → 175 out (120 reasoning)
git worktree add ../feature-wt feature printf "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:low__json_schema PASS 100% 192 in → 110 out (0 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qrfgp1bi/worktree_usage_f008 git worktree add ../feature-wt feature printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qrfgp1bi/worktree_usage_f008","git worktree add ../feature-wt feature","printf '%s' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:medium PASS 100% 142 in → 199 out (144 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:medium__json_schema PASS 100% 184 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:none PASS 100% 138 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature printf "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
openai/gpt-5.4-nano:none__json_schema PASS 100% 181 in → 109 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_b3yzxpeg/worktree_usage_f008 worktree add ../feature-wt feature printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_b3yzxpeg/worktree_usage_f008 worktree add ../feature-wt feature","printf '%s' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.4-nano:xhigh PASS 100% 140 in → 2,009 out (1,954 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 186 in → 674 out (610 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
openai/gpt-5.5:high__json_schema PASS 100% 187 in → 580 out (516 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.5:medium PASS 100% 143 in → 335 out (233 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0h1f3ii1/worktree_usage_f008 git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
openai/gpt-5.5:none__json_schema PASS 100% 185 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.6-luna:high__json_schema PASS 100% 187 in → 306 out (242 reasoning)
git worktree add ../feature-wt feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
openai/gpt-5.6-sol:none__json_schema PASS 100% 186 in → 67 out (0 reasoning)
git worktree add ../feature-wt feature printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-m.1:high__json_schema PASS 100% 170 in → 1,712 out (1,655 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-m.1:xhigh PASS 100% 171 in → 1,172 out (1,121 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-m.1:xhigh__json_schema PASS 100% 164 in → 1,230 out (1,155 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
poolside/laguna-xs-2.1:low PASS 100% 167 in → 835 out (785 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 166 in → 472 out (413 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
poolside/laguna-xs-2.1:none PASS 100% 171 in → 88 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file' </think>git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
poolside/laguna-xs-2.1:none__json_schema PASS 100% 166 in → 74 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
poolside/laguna-xs.2:high PASS 100% 205 in → 688 out (636 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-xs.2:low__json_schema PASS 100% 204 in → 259 out (200 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""]}
poolside/laguna-xs.2:medium PASS 100% 204 in → 1,058 out (1,004 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
poolside/laguna-xs.2:none__json_schema PASS 100% 203 in → 58 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'"]}
poolside/laguna-xs.2:xhigh PASS 100% 207 in → 289 out (238 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-27b:high__json_schema PASS 100% 171 in → 2,222 out (1,946 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
qwen/qwen3.6-27b:medium PASS 100% 170 in → 1,935 out (1,701 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-27b:none PASS 100% 167 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-27b:none__json_schema PASS 100% 168 in → 74 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'" ] }
qwen/qwen3.6-35b-a3b:high PASS 100% 167 in → 2,637 out (2,309 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 169 in → 2,449 out (2,151 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
qwen/qwen3.6-35b-a3b:none PASS 100% 172 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 173 in → 123 out (0 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_37s173jh/worktree_usage_f008 && git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_37s173jh/worktree_usage_f008 && git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'" ] }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 167 in → 2,663 out (2,457 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 170 in → 2,128 out (1,896 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
qwen/qwen3.6-flash:low PASS 100% 166 in → 2,591 out (2,537 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-flash:none PASS 100% 172 in → 59 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt checkout feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.6-flash:xhigh PASS 100% 172 in → 2,183 out (2,129 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-flash:xhigh PASS 100% 170 in → 1,765 out (1,711 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-max:none PASS 100% 173 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-max:xhigh PASS 100% 166 in → 1,937 out (1,883 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
qwen/qwen3.7-plus:high PASS 100% 168 in → 404 out (350 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
qwen/qwen3.7-plus:medium PASS 100% 171 in → 2,385 out (2,331 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
tencent/hy3:none__json_schema PASS 100% 151 in → 62 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:high__json_schema PASS 100% 154 in → 669 out (596 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
thinkingmachines/inkling-small:low PASS 100% 150 in → 117 out (62 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:low__json_schema PASS 100% 146 in → 77 out (5 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:max PASS 100% 150 in → 466 out (411 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:max__json_schema PASS 100% 150 in → 423 out (351 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling-small:medium PASS 100% 148 in → 476 out (409 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling-small:medium__json_schema PASS 100% 147 in → 353 out (289 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
thinkingmachines/inkling-small:none PASS 100% 144 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 151 in → 602 out (531 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:high PASS 100% 150 in → 452 out (397 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
thinkingmachines/inkling:low PASS 100% 153 in → 111 out (56 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:low__json_schema PASS 100% 150 in → 138 out (67 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:max PASS 100% 154 in → 1,927 out (1,872 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:medium PASS 100% 153 in → 178 out (122 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
thinkingmachines/inkling:none__json_schema PASS 100% 155 in → 63 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
thinkingmachines/inkling:xhigh__json_schema PASS 100% 149 in → 400 out (329 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:high__json_schema PASS 100% 382 in → 1,786 out (1,720 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:low PASS 100% 321 in → 588 out (539 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:low__json_schema PASS 100% 386 in → 748 out (695 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:max PASS 100% 315 in → 636 out (587 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:max__json_schema PASS 100% 387 in → 1,784 out (1,718 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
x-ai/grok-4.3:medium PASS 100% 325 in → 1,135 out (1,086 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:medium__json_schema PASS 100% 390 in → 1,161 out (1,108 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:none PASS 100% 313 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
x-ai/grok-4.3:none__json_schema PASS 100% 383 in → 53 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
x-ai/grok-4.3:xhigh__json_schema PASS 100% 382 in → 1,795 out (1,729 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 148 in → 1,136 out (1,014 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
z-ai/glm-4.7-flash:none__json_schema PASS 100% 146 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt && git commit -m 'Add new file'"]}
z-ai/glm-5.2:low PASS 100% 145 in → 2,728 out (2,495 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
z-ai/glm-5.2:none PASS 100% 147 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
anthropic/claude-sonnet-5:high FAIL 50% 197 in → 96 out (22 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:low FAIL 50% 193 in → 80 out (0 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:low__json_schema FAIL 50% 433 in → 86 out (0 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:medium FAIL 50% 193 in → 119 out (31 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:none FAIL 50% 193 in → 80 out (0 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:none__json_schema FAIL 50% 427 in → 85 out (0 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:xhigh FAIL 50% 192 in → 115 out (27 reasoning)
``` git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 50% 430 in → 340 out (74 reasoning)
git worktree add ../feature-wt feature echo -n 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo -n 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}]
arcee-ai/trinity-mini:high FAIL 50% 144 in → 624 out (606 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:high__json_schema FAIL 50% 140 in → 1,646 out (1,782 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:low__json_schema FAIL 50% 143 in → 663 out (609 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:medium FAIL 50% 146 in → 1,414 out (1,443 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:high FAIL 50% 181 in → 1,775 out (1,523 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:xhigh FAIL 50% 184 in → 2,650 out (2,424 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:medium FAIL 50% 186 in → 467 out (354 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:none FAIL 50% 185 in → 47 out (0 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:none__json_schema FAIL 50% 182 in → 56 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","echo 'new file' > ../feature-wt/new.txt","cd ../feature-wt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:xhigh__json_schema FAIL 50% 188 in → 1,063 out (842 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:high FAIL 50% 171 in → 1,139 out (1,126 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/devstral-2512 FAIL 50% 167 in → 48 out
```bash git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:none FAIL 50% 176 in → 48 out (0 reasoning)
```sh git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:none FAIL 50% 179 in → 53 out (0 reasoning)
git worktree add '../feature-wt' HEAD touch '../feature-wt/new.txt' echo 'new file' > '../feature-wt/new.txt' cd '../feature-wt' git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:none FAIL 50% 185 in → 44 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:low FAIL 50% 205 in → 75 out (22 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:high FAIL 50% 170 in → 1,101 out (1,055 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:medium FAIL 50% 172 in → 668 out (624 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 50% 164 in → 736 out (683 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:high__json_schema FAIL 50% 205 in → 203 out (136 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt git add ../feature-wt/new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo \"new file\" > ../feature-wt/new.txt", "git add ../feature-wt/new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:none FAIL 50% 210 in → 45 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:low FAIL 50% 173 in → 2,044 out (1,864 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:xhigh__json_schema FAIL 50% 173 in → 2,588 out (1 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:low FAIL 50% 170 in → 2,909 out (2,861 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:medium__json_schema FAIL 50% 172 in → 2,043 out (1,823 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-flash:high FAIL 50% 167 in → 1,668 out (1,620 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-flash:medium FAIL 50% 166 in → 1,725 out (1,677 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-flash:none FAIL 50% 169 in → 43 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.7-plus:none FAIL 50% 175 in → 43 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
thinkingmachines/inkling:none FAIL 50% 153 in → 52 out (0 reasoning)
git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'<|end_message|>
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:low FAIL 50% 147 in → 1,062 out (992 reasoning)
git worktree add ../feature-wt feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:xhigh FAIL 50% 149 in → 2,051 out (2,022 reasoning)
git worktree add ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:high FAIL 0% 193 in → 109 out (31 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:low FAIL 0% 198 in → 81 out (0 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:max FAIL 0% 201 in → 683 out (154 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:medium FAIL 0% 192 in → 104 out (22 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-fable-5:xhigh FAIL 0% 197 in → 520 out (185 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-haiku-4.5:xhigh FAIL 0% 197 in → 9,852 out (3,397 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 350 in → 120 out (31 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:max FAIL 0% 169 in → 378 out (176 reasoning)
``` git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 352 in → 798 out (293 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:medium FAIL 0% 172 in → 67 out (0 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 348 in → 120 out (31 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.7:max FAIL 0% 200 in → 389 out (62 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:high FAIL 0% 201 in → 91 out (10 reasoning)
``` git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:low FAIL 0% 193 in → 81 out (0 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:max FAIL 0% 195 in → 531 out (40 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 428 in → 896 out (89 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:medium FAIL 0% 194 in → 87 out (15 reasoning)
``` git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:none FAIL 0% 195 in → 79 out (0 reasoning)
``` git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-opus-4.8:xhigh FAIL 0% 198 in → 77 out (0 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:high FAIL 0% 172 in → 375 out (177 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 350 in → 328 out (179 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:max FAIL 0% 171 in → 581 out (205 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 352 in → 817 out (208 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 166 in → 425 out (156 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 352 in → 390 out (194 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-large-thinking:high FAIL 0% 142 in → 1,129 out (1,090 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low FAIL 0% 139 in → 2,272 out (2,237 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:medium FAIL 0% 140 in → 1,195 out (1,168 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low FAIL 0% 140 in → 1,012 out (1,025 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:medium__json_schema FAIL 0% 147 in → 825 out (791 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:xhigh FAIL 0% 139 in → 759 out (726 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 142 in → 621 out (566 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:high FAIL 0% 146 in → 477 out (413 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:high__json_schema FAIL 0% 139 in → 883 out (757 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:low FAIL 0% 225 in → 2,678 out (2,626 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:low__json_schema FAIL 0% 142 in → 598 out (513 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:max FAIL 0% 233 in → 14,603 out (14,551 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:max__json_schema FAIL 0% 224 in → 596 out (524 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file\n' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:medium FAIL 0% 151 in → 364 out (425 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:medium__json_schema FAIL 0% 142 in → 674 out (617 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:none FAIL 0% 142 in → 42 out (0 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 0% 144 in → 62 out (0 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:xhigh FAIL 0% 230 in → 263 out (206 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 144 in → 316 out (252 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\""] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:high FAIL 0% 144 in → 600 out (557 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:high__json_schema FAIL 0% 251 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' cd -
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","cd ../feature-wt","echo 'new file' > new.txt","git add new.txt","git commit -m 'Add new file'","cd -"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:low__json_schema FAIL 0% 144 in → 699 out (623 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:medium FAIL 0% 143 in → 664 out (621 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:none FAIL 0% 147 in → 44 out (0 reasoning)
```bash git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:xhigh FAIL 0% 222 in → 1,555 out (1,502 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-flash:xhigh__json_schema FAIL 0% 224 in → 1,149 out (1,106 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:high FAIL 0% 142 in → 793 out (751 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:low FAIL 0% 143 in → 603 out (550 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:low__json_schema FAIL 0% 148 in → 528 out (416 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_uifjpqq8/worktree_usage_f008 git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_uifjpqq8/worktree_usage_f008", "git worktree add ../feature-wt -b feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:medium FAIL 0% 144 in → 879 out (801 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:none FAIL 0% 142 in → 40 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 142 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:xhigh FAIL 0% 143 in → 1,072 out (1,019 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 418 in → 4,230 out (4,168 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:high FAIL 0% 167 in → 1,296 out (1,242 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:high__json_schema FAIL 0% 177 in → 1,133 out (1,075 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:low FAIL 0% 174 in → 881 out (827 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:medium FAIL 0% 169 in → 1,544 out (1,490 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:medium__json_schema FAIL 0% 171 in → 750 out (692 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:xhigh FAIL 0% 167 in → 1,854 out (1,800 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3-flash-preview:xhigh__json_schema FAIL 0% 251 in → 683 out (625 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:high FAIL 0% 171 in → 1,268 out (1,214 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 248 in → 1,174 out (1,112 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-flash-lite-preview:xhigh FAIL 0% 173 in → 1,944 out (1,890 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 250 in → 2,262 out (2,204 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:low FAIL 0% 172 in → 1,504 out (1,450 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 250 in → 843 out (785 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:medium FAIL 0% 171 in → 3,035 out (2,981 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.1-pro-preview:xhigh FAIL 0% 168 in → 2,914 out (2,860 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:high FAIL 0% 171 in → 1,433 out (1,379 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:xhigh FAIL 0% 175 in → 1,780 out (1,726 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 168 in → 1,115 out (1,081 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:high FAIL 0% 170 in → 2,196 out (2,153 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:high__json_schema FAIL 0% 241 in → 1,847 out (1,769 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:low FAIL 0% 170 in → 941 out (898 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:low__json_schema FAIL 0% 245 in → 593 out (515 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:medium FAIL 0% 169 in → 2,051 out (2,008 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:medium__json_schema FAIL 0% 247 in → 1,111 out (1,032 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:xhigh FAIL 0% 170 in → 1,258 out (1,204 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.5-flash:xhigh__json_schema FAIL 0% 246 in → 1,236 out (1,157 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:high FAIL 0% 174 in → 1,994 out (1,940 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:high__json_schema FAIL 0% 174 in → 3,144 out (3,111 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:low FAIL 0% 175 in → 743 out (689 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:low__json_schema FAIL 0% 171 in → 55 out (0 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:max__json_schema FAIL 0% 171 in → 1,697 out (1,663 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:medium FAIL 0% 172 in → 2,342 out (2,288 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:medium__json_schema FAIL 0% 244 in → 706 out (648 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:xhigh FAIL 0% 174 in → 2,830 out (2,776 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemini-3.6-flash:xhigh__json_schema FAIL 0% 172 in → 1,284 out (1,250 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo \"new file\" > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:high__json_schema FAIL 0% 186 in → 3,118 out (2,614 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:low FAIL 0% 185 in → 4,419 out (3,748 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-26b-a4b-it:low__json_schema FAIL 0% 183 in → 2,120 out (1,724 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
Failure: Expecting value: line 255 column 1 (char 1397)
google/gemma-4-31b-it:high__json_schema FAIL 0% 187 in → 1,742 out (1,465 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:low__json_schema FAIL 0% 179 in → 1,842 out (1,563 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:medium__json_schema FAIL 0% 186 in → 2,187 out (2,127 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
google/gemma-4-31b-it:xhigh FAIL 0% 188 in → 2,200 out (1,863 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
ibm-granite/granite-4.1-8b FAIL 0% 158 in → 54 out
```bash git worktree add ../feature-wt main echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
git worktree add ../feature-wt main echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt main", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
inclusionai/ling-2.6-flash FAIL 0% 177 in → 40 out
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
liquid/lfm-2-24b-a2b FAIL 0% 162 in → 43 out
```bash cd ../feature-wt git worktree add main feature echo "new file" > new.txt git add new.txt git commit -m "Add new file" ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: ``` cd ../feature-wt git worktree add ../feature feature echo "new file" > new.txt git add new.txt git commit -m "Add new file" ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:high__json_schema FAIL 0% 174 in → 684 out (602 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:low FAIL 0% 169 in → 2,020 out (1,954 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:low__json_schema FAIL 0% 187 in → 989 out (928 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:medium FAIL 0% 173 in → 1,456 out (1,392 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:medium__json_schema FAIL 0% 177 in → 928 out (875 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:xhigh FAIL 0% 166 in → 1,874 out (1,827 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.5:xhigh__json_schema FAIL 0% 173 in → 1,910 out (1,791 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.7:high__json_schema FAIL 0% 323 in → 1,592 out (1,521 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m2.7:medium FAIL 0% 173 in → 1,228 out (1,156 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:high FAIL 0% 307 in → 1,130 out (1,085 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:high__json_schema FAIL 0% 296 in → 70 out (0 reasoning)
git worktree add -b feature ../feature-wt HEAD echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt HEAD", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:low FAIL 0% 310 in → 1,409 out (1,336 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:low__json_schema FAIL 0% 261 in → 59 out (58 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt && git switch feature && echo 'new file' > new.txt && git add new.txt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "cd ../feature-wt && git switch feature && echo 'new file' > new.txt && git add new.txt && git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:medium FAIL 0% 315 in → 518 out (491 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:medium__json_schema FAIL 0% 299 in → 79 out (0 reasoning)
git worktree add -b feature ../feature-wt touch ../feature-wt/new.txt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "touch ../feature-wt/new.txt", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:xhigh FAIL 0% 309 in → 735 out (647 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
minimax/minimax-m3:xhigh__json_schema FAIL 0% 257 in → 100 out (99 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mmjbsbu9/worktree_usage_f008 git worktree add -b feature ../feature-wt cd ../feature-wt touch new.txt printf 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mmjbsbu9/worktree_usage_f008", "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "touch new.txt", "printf 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:high FAIL 0% 181 in → 3,552 out (3,800 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:low FAIL 0% 179 in → 3,616 out (3,838 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:medium FAIL 0% 181 in → 3,269 out (3,345 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 167 in → 2,192 out (2,279 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:none__json_schema FAIL 0% 164 in → 50 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:xhigh FAIL 0% 181 in → 2,641 out (2,708 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 166 in → 2,690 out (2,714 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:high FAIL 0% 140 in → 1,928 out (1,875 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:high__json_schema FAIL 0% 135 in → 2,787 out (2,812 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:low FAIL 0% 139 in → 3,111 out (3,110 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:low__json_schema FAIL 0% 142 in → 3,369 out (2,810 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:medium FAIL 0% 143 in → 3,189 out (3,150 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
moonshotai/kimi-k2.6:xhigh FAIL 0% 137 in → 1,627 out (1,598 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 140 in → 4,933 out (4,870 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": ["git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:high FAIL 0% 143 in → 2,911 out (2,858 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:low FAIL 0% 140 in → 528 out (481 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt && git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0% 142 in → 387 out (320 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:medium FAIL 0% 136 in → 798 out (793 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:medium__json_schema FAIL 0% 138 in → 348 out (298 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 137 in → 1,106 out (1,121 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k2.7-code:xhigh__json_schema FAIL 0% 141 in → 680 out (638 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:high FAIL 0% 214 in → 899 out (833 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:high__json_schema FAIL 0% 215 in → 784 out (718 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:low FAIL 0% 223 in → 785 out (719 reasoning)
git worktree add ../feature-wt -b feature echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:low__json_schema FAIL 0% 215 in → 1,027 out (961 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:max FAIL 0% 217 in → 1,063 out (997 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:max__json_schema FAIL 0% 216 in → 639 out (573 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
moonshotai/kimi-k3:medium__json_schema FAIL 0% 216 in → 1,121 out (1,055 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:low__json_schema FAIL 0% 181 in → 500 out (454 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh FAIL 0% 179 in → 459 out (439 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 185 in → 600 out (576 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:high FAIL 0% 182 in → 390 out (347 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:high__json_schema FAIL 0% 182 in → 278 out (218 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:low FAIL 0% 184 in → 335 out (289 reasoning)
git worktree add ../feature-wt -b feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:low__json_schema FAIL 0% 181 in → 262 out (197 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:medium FAIL 0% 180 in → 497 out (468 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:medium__json_schema FAIL 0% 181 in → 402 out (331 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:none__json_schema FAIL 0% 180 in → 59 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:xhigh FAIL 0% 181 in → 206 out (162 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema FAIL 0% 180 in → 417 out (356 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:high FAIL 0% 147 in → 691 out (516 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/worktree_usage_f008 && git worktree add -b feature ../feature-wt printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/feature-wt/new.txt cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_t2_qh0j6/feature-wt && git add new.txt && git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:high__json_schema FAIL 0% 188 in → 582 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:low FAIL 0% 141 in → 115 out (58 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:low__json_schema FAIL 0% 184 in → 148 out (80 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt && printf 'new file' > new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt && printf 'new file' > new.txt","cd ../feature-wt && git add new.txt","cd ../feature-wt && git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:medium FAIL 0% 142 in → 468 out (411 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:medium__json_schema FAIL 0% 188 in → 311 out (245 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:none FAIL 0% 145 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:none__json_schema FAIL 0% 186 in → 219 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/worktree_usage_f008 worktree add ../feature-wt -b feature printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/worktree_usage_f008 worktree add ../feature-wt -b feature","printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt/new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt add new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_8nm569ep/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:xhigh FAIL 0% 138 in → 27,516 out (27,282 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/worktree_usage_f008 worktree add -b feature ../feature-wt d4051ae41985470992f5db41630baf3c29b32123 printf 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt add -- new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_09yr34mt/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4-mini:xhigh__json_schema FAIL 0% 186 in → 25,124 out (24,862 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/worktree_usage_f008 worktree add -b feature /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt HEAD printf '%s' 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt/new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt add new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/worktree_usage_f008 worktree add -b feature /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt HEAD","printf '%s' 'new file' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt/new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt add new.txt && git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_o64_yueu/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:high FAIL 0% 141 in → 619 out (516 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_zlckm3pc/worktree_usage_f008 git worktree add -b feature ../feature-wt HEAD printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:high__json_schema FAIL 0% 187 in → 250 out (183 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:low FAIL 0% 141 in → 285 out (68 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_ws2lq3nf/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:low__json_schema FAIL 0% 191 in → 128 out (61 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:medium FAIL 0% 142 in → 461 out (360 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qb991ew1/worktree_usage_f008 git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:medium__json_schema FAIL 0% 191 in → 204 out (137 reasoning)
git worktree add -b feature ../feature-wt printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:none FAIL 0% 140 in → 182 out (0 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_n4j84znx/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:none__json_schema FAIL 0% 189 in → 65 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file ' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt -b feature","printf 'new file\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:xhigh FAIL 0% 141 in → 275 out (217 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.4:xhigh__json_schema FAIL 0% 185 in → 9,411 out (9,345 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:high FAIL 0% 142 in → 1,655 out (1,552 reasoning)
cd /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_mq4g8ke6/worktree_usage_f008 git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:low FAIL 0% 146 in → 208 out (150 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:low__json_schema FAIL 0% 189 in → 139 out (73 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:medium__json_schema FAIL 0% 187 in → 426 out (360 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:none FAIL 0% 147 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:xhigh FAIL 0% 141 in → 574 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.5:xhigh__json_schema FAIL 0% 186 in → 582 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:high FAIL 0% 139 in → 176 out (119 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:low FAIL 0% 141 in → 133 out (76 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:low__json_schema FAIL 0% 193 in → 183 out (117 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:max FAIL 0% 145 in → 305 out (248 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:max__json_schema FAIL 0% 189 in → 1,100 out (1,034 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:medium FAIL 0% 143 in → 204 out (147 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:medium__json_schema FAIL 0% 188 in → 204 out (138 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:none FAIL 0% 141 in → 55 out (0 reasoning)
git worktree add ../feature-wt -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:none__json_schema FAIL 0% 188 in → 64 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:xhigh FAIL 0% 143 in → 270 out (213 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-luna:xhigh__json_schema FAIL 0% 185 in → 289 out (223 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:high FAIL 0% 140 in → 179 out (119 reasoning)
git worktree add -b feature ../feature-wt printf '%s' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:high__json_schema FAIL 0% 190 in → 584 out (357 reasoning)
git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/worktree_usage_f008 worktree add -b feature ../feature-wt printf 'new file\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt/new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt add new.txt git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/worktree_usage_f008 worktree add -b feature ../feature-wt","printf 'new file\\n' > /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt/new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt add new.txt","git -C /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_0_yb62gi/feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:low FAIL 0% 137 in → 123 out (62 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:low__json_schema FAIL 0% 191 in → 140 out (69 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:max FAIL 0% 140 in → 573 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:max__json_schema FAIL 0% 188 in → 584 out (516 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:medium FAIL 0% 141 in → 102 out (41 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:medium__json_schema FAIL 0% 188 in → 165 out (97 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:none FAIL 0% 138 in → 59 out (0 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:xhigh FAIL 0% 139 in → 188 out (131 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-sol:xhigh__json_schema FAIL 0% 185 in → 273 out (202 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:high FAIL 0% 146 in → 150 out (92 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:high__json_schema FAIL 0% 185 in → 162 out (94 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:low FAIL 0% 140 in → 115 out (57 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:low__json_schema FAIL 0% 189 in → 116 out (48 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:max FAIL 0% 137 in → 389 out (331 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:max__json_schema FAIL 0% 187 in → 282 out (211 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:medium FAIL 0% 141 in → 143 out (85 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:medium__json_schema FAIL 0% 190 in → 212 out (141 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:none FAIL 0% 140 in → 56 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:none__json_schema FAIL 0% 187 in → 66 out (0 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file\\n' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:xhigh FAIL 0% 138 in → 459 out (398 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-5.6-terra:xhigh__json_schema FAIL 0% 184 in → 208 out (137 reasoning)
git worktree add -b feature ../feature-wt printf '%s\n' 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf '%s\\n' 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:high FAIL 0% 200 in → 1,523 out (1,565 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:high__json_schema FAIL 0% 201 in → 814 out (750 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:low FAIL 0% 198 in → 88 out (43 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:low__json_schema FAIL 0% 205 in → 205 out (156 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt printf 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt","printf 'new file' > new.txt","git add new.txt","git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:medium FAIL 0% 204 in → 330 out (271 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:medium__json_schema FAIL 0% 189 in → 415 out (331 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:xhigh FAIL 0% 185 in → 1,230 out (1,276 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-120b:xhigh__json_schema FAIL 0% 204 in → 1,484 out (1,507 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","printf 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:high FAIL 0% 201 in → 1,467 out (1,513 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:high__json_schema FAIL 0% 190 in → 596 out (554 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","cd ../feature-wt","echo \"new file\" > new.txt","git add new.txt","git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:low__json_schema FAIL 0% 200 in → 97 out (43 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add ../feature-wt feature","cd ../feature-wt","echo 'new file' > new.txt","git add new.txt","git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:medium FAIL 0% 200 in → 582 out (543 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:medium__json_schema FAIL 0% 197 in → 343 out (281 reasoning)
git worktree add -b feature ../feature-wt cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:xhigh FAIL 0% 202 in → 1,282 out (1,234 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 189 in → 549 out (485 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo "new file" > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo \"new file\" > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:low FAIL 0% 167 in → 1,089 out (1,049 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:low__json_schema FAIL 0% 169 in → 2,624 out (2,551 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file' cd -
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'", "cd -" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:medium FAIL 0% 165 in → 1,208 out (1,166 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:medium__json_schema FAIL 0% 171 in → 2,367 out (2,300 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:none FAIL 0% 172 in → 63 out (0 reasoning)
git worktree add ../feature-wt feature git -C ../feature-wt checkout -b feature printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-m.1:none__json_schema FAIL 0% 172 in → 49 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:high FAIL 0% 171 in → 1,728 out (1,688 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:high__json_schema FAIL 0% 173 in → 1,215 out (1,148 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:low__json_schema FAIL 0% 167 in → 772 out (723 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs-2.1:xhigh FAIL 0% 165 in → 815 out (775 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:low FAIL 0% 204 in → 783 out (727 reasoning)
git branch feature git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:medium__json_schema FAIL 0% 204 in → 2,056 out (2,001 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt", "echo \"new file\" > ../feature-wt/new.txt", "cd ../feature-wt", "git add new.txt", "git commit -m \"Add new file\""]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
poolside/laguna-xs.2:xhigh__json_schema FAIL 0% 205 in → 411 out (349 reasoning)
git worktree add -b feature ../feature-wt main echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add new.txt cd ../feature-wt && git commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands": ["git worktree add -b feature ../feature-wt main", "echo 'new file' > ../feature-wt/new.txt", "cd ../feature-wt && git add new.txt", "cd ../feature-wt && git commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:high FAIL 0% 169 in → 2,452 out (2,181 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 173 in → 2,218 out (1,974 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m \"Add new file\"" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-27b:xhigh FAIL 0% 171 in → 3,700 out (3,650 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:low__json_schema FAIL 0% 164 in → 2,048 out (1,832 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt feature", "cd ../feature-wt", "echo 'new file' > new.txt", "git add new.txt", "git commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
qwen/qwen3.6-35b-a3b:medium FAIL 0% 168 in → 3,358 out (2,946 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abe29d1b-c2ff-9b55-8b53-d704ddfa9d3a","request_id":"abe29d1b-c2ff-9b55-8b53-d704ddfa9d3a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6eeedb1-9704-921b-a1bb-95583a73de4e","request_id":"e6eeedb1-9704-921b-a1bb-95583a73de4e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium FAIL 0% 169 in → 1,913 out (1,857 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2e73abef-e4ce-9a98-9e87-5b220289f0e7","request_id":"2e73abef-e4ce-9a98-9e87-5b220289f0e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bf8874e1-8a6b-9276-aee2-1a03c1fa2e24","request_id":"bf8874e1-8a6b-9276-aee2-1a03c1fa2e24"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4fc7bbbb-0d7a-978b-aab3-81b681d29881","request_id":"4fc7bbbb-0d7a-978b-aab3-81b681d29881"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high FAIL 0% 168 in → 2,087 out (2,037 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4a27c070-d19e-922c-aa91-a5bd10610acc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low FAIL 0% 168 in → 2,276 out (2,227 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt && git add . && git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-dd9f1329-0efc-92b9-951f-9d25dd0d1502"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 171 in → 2,030 out (1,980 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5f5fd8f2-d7e4-93c4-9b25-f969fc670da0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9e5d9abe-23d7-9033-bd26-72a3025866a6"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6621c924-107b-9c6a-adbf-e284575e66fb"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-411db393-eb88-9a61-b39a-c604711b8f6e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high FAIL 0% 170 in → 1,188 out (1,132 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-06294a1b-49ee-9153-8d5c-aab2ed96261f","request_id":"06294a1b-49ee-9153-8d5c-aab2ed96261f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low FAIL 0% 174 in → 1,381 out (1,325 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-90e71462-fd89-96a3-9d2a-1da6baac9533","request_id":"90e71462-fd89-96a3-9d2a-1da6baac9533"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium FAIL 0% 167 in → 1,735 out (1,679 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6c08186-7ca9-98c4-af7f-5cca12529b3e","request_id":"a6c08186-7ca9-98c4-af7f-5cca12529b3e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fbde3990-7d19-957f-a366-ebd0bc041c9f","request_id":"fbde3990-7d19-957f-a366-ebd0bc041c9f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f3578958-71e2-9214-95ba-dd354e046d4a","request_id":"f3578958-71e2-9214-95ba-dd354e046d4a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2ff7de57-3e0d-916e-9a88-7b88f31fced8","request_id":"2ff7de57-3e0d-916e-9a88-7b88f31fced8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low FAIL 0% 167 in → 2,111 out (2,055 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d22ee697-4e11-90b1-81e9-ce7978a4a3e6","request_id":"d22ee697-4e11-90b1-81e9-ce7978a4a3e6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-4cca6aa0-c616-982d-ba32-0710b64b9c08","request_id":"4cca6aa0-c616-982d-ba32-0710b64b9c08"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a56e14e2-cabc-93ef-bf2b-39f8910c36e2","request_id":"a56e14e2-cabc-93ef-bf2b-39f8910c36e2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:xhigh FAIL 0% 171 in → 2,053 out (1,997 reasoning)
git worktree add -b feature ../feature-wt echo "new file" > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-873573ec-17cf-9204-ae59-b4c4c729d79b","request_id":"873573ec-17cf-9204-ae59-b4c4c729d79b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
tencent/hy3:high FAIL 0% 154 in → 5,439 out (5,386 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:high__json_schema FAIL 0% 152 in → 8,332 out (8,279 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:low FAIL 0% 149 in → 860 out (807 reasoning)
git worktree add -b feature ../feature-wt printf 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:low__json_schema FAIL 0% 149 in → 1,427 out (1,276 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:medium FAIL 0% 152 in → 1,787 out (1,740 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt cd ../feature-wt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:medium__json_schema FAIL 0% 153 in → 1,557 out (1,379 reasoning)
git worktree add -b feature ../feature-wt printf 'new file\n' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "printf 'new file\\n' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:none FAIL 0% 155 in → 40 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:xhigh FAIL 0% 151 in → 11,354 out (11,301 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
tencent/hy3:xhigh__json_schema FAIL 0% 154 in → 3,584 out (3,531 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
thinkingmachines/inkling-small:high FAIL 0% 150 in → 671 out (630 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'{"commands": ["git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'"]}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling-small:xhigh FAIL 0% 153 in → 923 out (866 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:xhigh FAIL 0% 151 in → 648 out (590 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.3:high FAIL 0% 318 in → 4,613 out (4,562 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.3:xhigh FAIL 0% 317 in → 2,462 out (2,411 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:high FAIL 0% 352 in → 701 out (649 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:high__json_schema FAIL 0% 431 in → 1,019 out (962 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:low FAIL 0% 353 in → 371 out (316 reasoning)
``` git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file' ```
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:low__json_schema FAIL 0% 430 in → 621 out (564 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:max FAIL 0% 355 in → 1,536 out (1,484 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:max__json_schema FAIL 0% 431 in → 1,476 out (1,403 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:medium FAIL 0% 350 in → 728 out (676 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:medium__json_schema FAIL 0% 428 in → 1,454 out (1,397 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) {"commands":["git worktree add -b feature ../feature-wt","echo 'new file' > ../feature-wt/new.txt","git -C ../feature-wt add new.txt","git -C ../feature-wt commit -m 'Add new file'"]}
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:xhigh FAIL 0% 355 in → 1,107 out (1,055 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
x-ai/grok-4.5:xhigh__json_schema FAIL 0% 429 in → 1,499 out (1,426 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:high FAIL 0% 143 in → 1,646 out (1,471 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:medium FAIL 0% 148 in → 1,228 out (1,201 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-4.7-flash:none FAIL 0% 145 in → 39 out (0 reasoning)
git worktree add ../feature-wt feature cd ../feature-wt echo 'new file' > new.txt git add new.txt git commit -m "Add new file"
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:high FAIL 0% 144 in → 3,260 out (2,974 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:high__json_schema FAIL 0% 158 in → 992 out (981 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "commands": [ ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:low__json_schema FAIL 0% 150 in → 788 out (724 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add -b feature ../feature-wt", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:medium FAIL 0% 155 in → 203 out (150 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:medium__json_schema FAIL 0% 338 in → 498 out (427 reasoning)
git worktree add ../feature-wt -b feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add ../feature-wt -b feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:none__json_schema FAIL 0% 145 in → 66 out (0 reasoning)
git worktree add '../feature-wt' -b feature echo 'new file' > '../feature-wt/new.txt' git -C '../feature-wt' add new.txt git -C '../feature-wt' commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "git worktree add '../feature-wt' -b feature", "echo 'new file' > '../feature-wt/new.txt'", "git -C '../feature-wt' add new.txt", "git -C '../feature-wt' commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:xhigh FAIL 0% 146 in → 567 out (438 reasoning)
git worktree add -b feature ../feature-wt echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]
z-ai/glm-5.2:xhigh__json_schema FAIL 0% 157 in → 1,240 out (1,122 reasoning)
Cd to /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qivxl_d6/worktree_usage_f008 git worktree add ../feature-wt feature echo 'new file' > ../feature-wt/new.txt git -C ../feature-wt add new.txt git -C ../feature-wt commit -m 'Add new file'
JSON Schema Structured Output
(raw) { "commands": [ "Cd to /private/var/folders/kg/gb4597jx5jv6b7tngk181jr40000gn/T/worktree_usage_f008_qivxl_d6/worktree_usage_f008", "git worktree add ../feature-wt feature", "echo 'new file' > ../feature-wt/new.txt", "git -C ../feature-wt add new.txt", "git -C ../feature-wt commit -m 'Add new file'" ] }
Failure: Failed assertions: [{'type': 'file_content', 'path': '../feature-wt/new.txt', 'value': 'new file'}, {'type': 'git_output', 'command': 'git -C ../feature-wt log --oneline -1', 'contains': 'Add new file'}]