Multi-line function body conflict
Tests ability to resolve a function-body conflict during rebase. Evaluates semantic merging of code with rebase direction awareness.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'def greet(name): print("Hello") return True' > greet.py
  5. 05 git add greet.py
  6. 06 git commit -m 'Initial greet'
  7. 07 git checkout -b feature
  8. 08 printf 'def greet(name): print(f"Hello, {name}") return True' > greet.py
  9. 09 git add greet.py
  10. 10 git commit -m 'Add name parameter'
  11. 11 git checkout main
  12. 12 printf 'def greet(name): print("Hi there") return False' > greet.py
  13. 13 git add greet.py
  14. 14 git commit -m 'Change greeting'
  15. 15 git rebase feature
Prompt
Resolve the rebase conflict in greet.py. The base was a simple greet with print("Hello"). Branch being rebased added f-string formatting with the name parameter. Main branch changed the greeting text to "Hi there" and changed return to False. Resolve by taking main branch's greeting text and False return rather than the rebased branch's name-aware greeting. Provide ONLY the resolved file content.
Expected
def greet(name): print("Hi there") return False

Scoped model quality, cost, API time, and token usage for rebase/f005.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 224 in → 57 out (26 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-fable-5:high__json_schema PASS 100% 443 in → 193 out (78 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-fable-5:low PASS 100% 224 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-fable-5:low__json_schema PASS 100% 445 in → 84 out (49 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-fable-5:max PASS 100% 225 in → 623 out (169 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-fable-5:max__json_schema PASS 100% 445 in → 779 out (179 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-fable-5:medium PASS 100% 226 in → 34 out (12 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 444 in → 133 out (79 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-fable-5:xhigh PASS 100% 223 in → 289 out (135 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 447 in → 202 out (48 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-haiku-4.5:high PASS 100% 191 in → 788 out (548 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 360 in → 473 out (419 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
anthropic/claude-haiku-4.5:low PASS 100% 191 in → 736 out (550 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 358 in → 587 out (517 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-haiku-4.5:medium PASS 100% 190 in → 1,689 out (811 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 359 in → 1,779 out (1,062 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-haiku-4.5:none PASS 100% 162 in → 25 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 327 in → 32 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 189 in → 720 out (543 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 357 in → 523 out (490 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:high PASS 100% 161 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 329 in → 112 out (73 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:low PASS 100% 162 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 329 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:max PASS 100% 160 in → 204 out (186 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:max__json_schema PASS 100% 329 in → 190 out (144 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:medium PASS 100% 161 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 329 in → 32 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:none PASS 100% 162 in → 25 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 328 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 161 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 329 in → 113 out (72 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:high PASS 100% 230 in → 431 out (134 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.7:high__json_schema PASS 100% 449 in → 41 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:low PASS 100% 229 in → 23 out (0 reasoning)
def greet(name): print("Hi there") return False
anthropic/claude-opus-4.7:low__json_schema PASS 100% 450 in → 41 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:max PASS 100% 229 in → 2,582 out (340 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 450 in → 2,175 out (223 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:medium PASS 100% 228 in → 23 out (0 reasoning)
def greet(name): print("Hi there") return False
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 449 in → 41 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:none PASS 100% 229 in → 29 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 451 in → 42 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 228 in → 645 out (157 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 450 in → 41 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:high PASS 100% 225 in → 403 out (127 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 445 in → 39 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:low PASS 100% 225 in → 242 out (111 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 443 in → 39 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:max PASS 100% 224 in → 1,031 out (141 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 445 in → 791 out (114 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:medium PASS 100% 225 in → 299 out (155 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 444 in → 39 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:none PASS 100% 226 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 443 in → 38 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 224 in → 773 out (150 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 447 in → 498 out (146 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 162 in → 177 out (132 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 330 in → 138 out (97 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 163 in → 25 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 331 in → 32 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 165 in → 674 out (361 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 331 in → 224 out (175 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 164 in → 25 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 329 in → 32 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 161 in → 25 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 331 in → 32 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 161 in → 891 out (372 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 329 in → 120 out (76 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-5:high PASS 100% 225 in → 148 out (47 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 444 in → 65 out (34 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-5:low PASS 100% 224 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 443 in → 39 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-5:medium PASS 100% 226 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 445 in → 123 out (34 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-5:none PASS 100% 225 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 444 in → 38 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 227 in → 26 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 445 in → 1,018 out (221 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 137 in → 273 out (253 reasoning)
```python def greet(name): print("Hi there") return False ```
arcee-ai/trinity-large-thinking:low PASS 100% 137 in → 538 out (524 reasoning)
def greet(name): print("Hi there") return False
arcee-ai/trinity-large-thinking:medium PASS 100% 136 in → 358 out (340 reasoning)
```python def greet(name): print("Hi there") return False ```
arcee-ai/trinity-large-thinking:xhigh PASS 100% 137 in → 253 out (238 reasoning)
def greet(name): print("Hi there") return False
arcee-ai/trinity-mini:high PASS 100% 137 in → 556 out (625 reasoning)
def greet(name): print("Hi there") return False
arcee-ai/trinity-mini:high__json_schema PASS 100% 139 in → 349 out (403 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
arcee-ai/trinity-mini:low PASS 100% 138 in → 305 out (352 reasoning)
def greet(name): print("Hi there") return False
arcee-ai/trinity-mini:low__json_schema PASS 100% 136 in → 457 out (510 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
arcee-ai/trinity-mini:medium__json_schema PASS 100% 136 in → 326 out (348 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
arcee-ai/trinity-mini:xhigh PASS 100% 139 in → 392 out (458 reasoning)
def greet(name): print("Hi there") return False
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 138 in → 375 out (403 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
deepseek/deepseek-v4-flash-0731:high PASS 100% 141 in → 166 out (165 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 139 in → 423 out (474 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 140 in → 254 out (240 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 285 in → 364 out (386 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
deepseek/deepseek-v4-flash-0731:max PASS 100% 219 in → 121 out (107 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 219 in → 1,133 out (1,231 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 139 in → 195 out (181 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 140 in → 326 out (344 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 146 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 142 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 140 in → 202 out (188 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 221 in → 134 out (128 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
deepseek/deepseek-v4-flash:high PASS 100% 140 in → 95 out (94 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 140 in → 169 out (138 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash:low PASS 100% 140 in → 151 out (132 reasoning)
```python def greet(name): print("Hi there") return False ```
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 143 in → 131 out (98 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-flash:medium PASS 100% 140 in → 160 out (147 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 238 in → 26 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
deepseek/deepseek-v4-flash:none PASS 100% 140 in → 19 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 140 in → 25 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 218 in → 2,332 out (2,683 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-pro:high PASS 100% 139 in → 196 out (181 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 333 in → 156 out (125 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-pro:low PASS 100% 140 in → 428 out (414 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 142 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
deepseek/deepseek-v4-pro:medium PASS 100% 141 in → 343 out (362 reasoning)
def greet(name): print("Hi there") return False
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 333 in → 763 out (737 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
deepseek/deepseek-v4-pro:none PASS 100% 140 in → 19 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
deepseek/deepseek-v4-pro:xhigh PASS 100% 219 in → 474 out (454 reasoning)
```python def greet(name): print("Hi there") return False ```
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 140 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
google/gemini-3-flash-preview:high PASS 100% 147 in → 351 out (335 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3-flash-preview:high__json_schema PASS 100% 204 in → 669 out (645 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3-flash-preview:low PASS 100% 147 in → 242 out (226 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3-flash-preview:low__json_schema PASS 100% 208 in → 708 out (683 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3-flash-preview:medium PASS 100% 147 in → 388 out (372 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3-flash-preview:medium__json_schema PASS 100% 205 in → 442 out (412 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3-flash-preview:xhigh PASS 100% 146 in → 614 out (598 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 205 in → 368 out (343 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 149 in → 1,550 out (1,534 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 208 in → 1,244 out (1,214 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3.1-flash-lite-preview:low PASS 100% 148 in → 141 out (125 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 207 in → 130 out (105 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 146 in → 281 out (265 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 147 in → 478 out (462 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 148 in → 2,616 out (2,592 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3.1-pro-preview:high PASS 100% 147 in → 606 out (590 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 206 in → 1,321 out (1,295 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.1-pro-preview:low PASS 100% 147 in → 444 out (428 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 206 in → 139 out (112 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.1-pro-preview:medium PASS 100% 148 in → 576 out (560 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 207 in → 710 out (684 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 145 in → 479 out (463 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 206 in → 1,204 out (1,179 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.5-flash-lite:high PASS 100% 149 in → 482 out (466 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 208 in → 2,840 out (2,816 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3.5-flash-lite:low PASS 100% 149 in → 16 out (0 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 206 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3.5-flash-lite:max PASS 100% 148 in → 783 out (767 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 207 in → 1,080 out (1,054 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.5-flash-lite:medium PASS 100% 148 in → 447 out (431 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 149 in → 268 out (238 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3.5-flash-lite:xhigh PASS 100% 145 in → 1,166 out (1,150 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 149 in → 1,160 out (1,135 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemini-3.5-flash:high PASS 100% 147 in → 646 out (630 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash:high__json_schema PASS 100% 207 in → 1,262 out (1,235 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.5-flash:low PASS 100% 146 in → 500 out (484 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash:low__json_schema PASS 100% 208 in → 332 out (305 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.5-flash:medium PASS 100% 148 in → 529 out (513 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash:medium__json_schema PASS 100% 208 in → 641 out (614 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.5-flash:xhigh PASS 100% 145 in → 837 out (821 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 207 in → 942 out (916 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.6-flash:high PASS 100% 147 in → 491 out (475 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.6-flash:high__json_schema PASS 100% 147 in → 1,086 out (1,060 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.6-flash:low PASS 100% 148 in → 266 out (250 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.6-flash:low__json_schema PASS 100% 149 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3.6-flash:max PASS 100% 147 in → 616 out (600 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.6-flash:max__json_schema PASS 100% 148 in → 421 out (391 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemini-3.6-flash:medium PASS 100% 149 in → 480 out (464 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.6-flash:medium__json_schema PASS 100% 149 in → 651 out (625 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemini-3.6-flash:xhigh PASS 100% 148 in → 449 out (433 reasoning)
def greet(name): print("Hi there") return False
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 149 in → 930 out (904 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
google/gemma-4-26b-a4b-it:high PASS 100% 163 in → 1,401 out (1,384 reasoning)
def greet(name): print("Hi there") return False
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 163 in → 566 out (525 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemma-4-26b-a4b-it:low PASS 100% 162 in → 1,393 out (1,239 reasoning)
def greet(name): print("Hi there") return False
google/gemma-4-26b-a4b-it:medium PASS 100% 161 in → 1,058 out (1,078 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 163 in → 1,965 out (1,990 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemma-4-26b-a4b-it:none PASS 100% 158 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 161 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 163 in → 1,142 out (1,090 reasoning)
def greet(name): print("Hi there") return False
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 162 in → 1,414 out (1,412 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
google/gemma-4-31b-it:high PASS 100% 164 in → 302 out (266 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-31b-it:high__json_schema PASS 100% 163 in → 424 out (402 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemma-4-31b-it:low PASS 100% 163 in → 416 out (394 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-31b-it:low__json_schema PASS 100% 162 in → 417 out (372 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
google/gemma-4-31b-it:medium PASS 100% 164 in → 336 out (286 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-31b-it:medium__json_schema PASS 100% 163 in → 420 out (378 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemma-4-31b-it:none PASS 100% 160 in → 22 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-31b-it:none__json_schema PASS 100% 159 in → 26 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
google/gemma-4-31b-it:xhigh PASS 100% 162 in → 837 out (795 reasoning)
```python def greet(name): print("Hi there") return False ```
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 163 in → 268 out (214 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
ibm-granite/granite-4.1-8b PASS 100% 150 in → 19 out
```python def greet(name): print("Hi there") return False ```
ibm-granite/granite-4.1-8b__json_schema PASS 100% 151 in → 28 out
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
inclusionai/ling-2.6-flash PASS 100% 159 in → 16 out
def greet(name): print("Hi there") return False
inclusionai/ling-2.6-flash__json_schema PASS 100% 161 in → 27 out
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
liquid/lfm-2-24b-a2b PASS 100% 156 in → 19 out
``` def greet(name): print("Hi there") return False ```
minimax/minimax-m2.5:high PASS 100% 170 in → 261 out (286 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.5:high__json_schema PASS 100% 186 in → 494 out (575 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
minimax/minimax-m2.5:low PASS 100% 170 in → 489 out (599 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.5:low__json_schema PASS 100% 169 in → 481 out (540 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
minimax/minimax-m2.5:medium PASS 100% 170 in → 310 out (356 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.5:medium__json_schema PASS 100% 186 in → 534 out (504 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
minimax/minimax-m2.5:xhigh PASS 100% 171 in → 195 out (180 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 184 in → 339 out (307 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
minimax/minimax-m2.7:high PASS 100% 170 in → 520 out (620 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.7:low PASS 100% 175 in → 336 out (322 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.7:medium PASS 100% 170 in → 370 out (354 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m2.7:medium__json_schema PASS 100% 305 in → 1,015 out (984 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
minimax/minimax-m2.7:xhigh PASS 100% 171 in → 249 out (233 reasoning)
def greet(name): print("Hi there") return False
minimax/minimax-m3:high__json_schema PASS 100% 295 in → 27 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
minimax/minimax-m3:low PASS 100% 309 in → 350 out (369 reasoning)
```python def greet(name): print("Hi there") return False ```
minimax/minimax-m3:medium PASS 100% 307 in → 160 out (158 reasoning)
```python def greet(name): print("Hi there") return False ```
minimax/minimax-m3:medium__json_schema PASS 100% 294 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
minimax/minimax-m3:xhigh PASS 100% 307 in → 183 out (184 reasoning)
```python def greet(name): print("Hi there") return False ```
minimax/minimax-m3:xhigh__json_schema PASS 100% 295 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
mistralai/devstral-2512__json_schema PASS 100% 144 in → 27 out
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
mistralai/mistral-medium-3-5:high PASS 100% 155 in → 602 out (675 reasoning)
def greet(name): print("Hi there") return False
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 144 in → 224 out (225 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
mistralai/mistral-medium-3-5:low PASS 100% 153 in → 274 out (288 reasoning)
def greet(name): print("Hi there") return False
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 144 in → 256 out (261 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
mistralai/mistral-medium-3-5:medium PASS 100% 154 in → 476 out (522 reasoning)
def greet(name): print("Hi there") return False
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 144 in → 300 out (322 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
mistralai/mistral-medium-3-5:none PASS 100% 156 in → 19 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
mistralai/mistral-medium-3-5:xhigh PASS 100% 155 in → 1,155 out (1,322 reasoning)
def greet(name): print("Hi there") return False
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 144 in → 361 out (382 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
moonshotai/kimi-k2.6:low PASS 100% 141 in → 2,294 out (2,274 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k2.6:medium PASS 100% 139 in → 4,950 out (4,930 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k2.6:xhigh PASS 100% 139 in → 1,214 out (1,194 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 139 in → 1,620 out (1,843 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 138 in → 260 out (233 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
moonshotai/kimi-k2.7-code:medium PASS 100% 140 in → 531 out (511 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k2.7-code:xhigh PASS 100% 139 in → 730 out (715 reasoning)
def greet(name): print("Hi there") return False
moonshotai/kimi-k3:high PASS 100% 215 in → 562 out (534 reasoning)
def greet(name): print("Hi there") return False
moonshotai/kimi-k3:high__json_schema PASS 100% 216 in → 683 out (655 reasoning)
def greet(name): print("Hi there") return False
moonshotai/kimi-k3:low PASS 100% 217 in → 518 out (485 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k3:low__json_schema PASS 100% 218 in → 293 out (260 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k3:max PASS 100% 214 in → 440 out (407 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k3:max__json_schema PASS 100% 215 in → 476 out (448 reasoning)
def greet(name): print("Hi there") return False
moonshotai/kimi-k3:medium PASS 100% 217 in → 338 out (305 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k3:medium__json_schema PASS 100% 313 in → 259 out (223 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
moonshotai/kimi-k3:xhigh PASS 100% 217 in → 550 out (517 reasoning)
```python def greet(name): print("Hi there") return False ```
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 313 in → 118 out (82 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 157 in → 139 out (156 reasoning)
```python def greet(name): print("Hi there") return False ```
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 156 in → 222 out (234 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 155 in → 362 out (413 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 155 in → 183 out (192 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 155 in → 70 out (67 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 156 in → 91 out (81 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 156 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 156 in → 29 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 155 in → 158 out (172 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 156 in → 132 out (123 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 156 in → 659 out (615 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 157 in → 122 out (89 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 155 in → 232 out (211 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 156 in → 175 out (141 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 155 in → 71 out (55 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 155 in → 240 out (206 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 157 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 156 in → 30 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 155 in → 163 out (169 reasoning)
def greet(name): print("Hi there") return False
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 157 in → 103 out (71 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
openai/gpt-5.4-mini:high PASS 100% 134 in → 81 out (62 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-mini:high__json_schema PASS 100% 170 in → 125 out (93 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.4-mini:low PASS 100% 135 in → 48 out (29 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-mini:low__json_schema PASS 100% 171 in → 61 out (29 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.4-mini:medium PASS 100% 136 in → 71 out (52 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-mini:medium__json_schema PASS 100% 172 in → 83 out (50 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-mini:none PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-mini:none__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-mini:xhigh PASS 100% 134 in → 100 out (81 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 169 in → 129 out (97 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.4-nano:high PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-nano:high__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-nano:low PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-nano:low__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-nano:medium PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-nano:medium__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-nano:none PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4-nano:none__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 136 in → 121 out (97 reasoning)
```python def greet(name): print("Hi there") return False ```
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4:high PASS 100% 134 in → 59 out (40 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4:high__json_schema PASS 100% 172 in → 200 out (167 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4:low PASS 100% 136 in → 29 out (10 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4:low__json_schema PASS 100% 171 in → 42 out (9 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4:medium PASS 100% 133 in → 78 out (59 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4:medium__json_schema PASS 100% 170 in → 45 out (12 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4:none PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4:none__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.4:xhigh PASS 100% 135 in → 342 out (323 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.4:xhigh__json_schema PASS 100% 173 in → 142 out (110 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.5:high PASS 100% 135 in → 179 out (160 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.5:high__json_schema PASS 100% 173 in → 369 out (336 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.5:low PASS 100% 134 in → 63 out (44 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.5:low__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.5:medium PASS 100% 134 in → 90 out (71 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.5:medium__json_schema PASS 100% 171 in → 157 out (124 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.5:none PASS 100% 137 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.5:none__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.5:xhigh PASS 100% 137 in → 171 out (152 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.5:xhigh__json_schema PASS 100% 170 in → 294 out (261 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-luna:high PASS 100% 134 in → 57 out (38 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:high__json_schema PASS 100% 171 in → 76 out (44 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.6-luna:low PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:low__json_schema PASS 100% 170 in → 63 out (31 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.6-luna:max PASS 100% 137 in → 77 out (58 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:max__json_schema PASS 100% 171 in → 358 out (326 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.6-luna:medium PASS 100% 135 in → 54 out (35 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:medium__json_schema PASS 100% 171 in → 56 out (23 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-luna:none PASS 100% 136 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:none__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-luna:xhigh PASS 100% 135 in → 61 out (42 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 171 in → 160 out (128 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-5.6-sol:high PASS 100% 136 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:high__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-sol:low PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:low__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-sol:max PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:max__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-sol:medium PASS 100% 133 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:medium__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-sol:none PASS 100% 134 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:none__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-sol:xhigh PASS 100% 136 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:high PASS 100% 136 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:high__json_schema PASS 100% 172 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:low PASS 100% 138 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:low__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:max PASS 100% 136 in → 80 out (61 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:max__json_schema PASS 100% 170 in → 549 out (516 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:medium PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:medium__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:none PASS 100% 135 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:none__json_schema PASS 100% 170 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-5.6-terra:xhigh PASS 100% 136 in → 17 out (0 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 171 in → 31 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-oss-120b:high PASS 100% 193 in → 229 out (268 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-120b:high__json_schema PASS 100% 185 in → 491 out (551 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
openai/gpt-oss-120b:low PASS 100% 198 in → 120 out (130 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-120b:low__json_schema PASS 100% 196 in → 71 out (34 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
openai/gpt-oss-120b:medium PASS 100% 183 in → 109 out (105 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-120b:medium__json_schema PASS 100% 199 in → 170 out (179 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-oss-120b:xhigh PASS 100% 199 in → 516 out (620 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 184 in → 391 out (452 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
openai/gpt-oss-20b:high PASS 100% 193 in → 590 out (681 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-20b:high__json_schema PASS 100% 194 in → 381 out (450 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-oss-20b:low PASS 100% 199 in → 85 out (72 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-20b:low__json_schema PASS 100% 199 in → 67 out (40 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False\n"}
openai/gpt-oss-20b:medium PASS 100% 196 in → 158 out (176 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-20b:medium__json_schema PASS 100% 197 in → 190 out (159 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
openai/gpt-oss-20b:xhigh PASS 100% 195 in → 318 out (297 reasoning)
def greet(name): print("Hi there") return False
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 299 in → 808 out (773 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
poolside/laguna-m.1:high PASS 100% 151 in → 331 out (313 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-m.1:high__json_schema PASS 100% 149 in → 652 out (628 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
poolside/laguna-m.1:low PASS 100% 150 in → 800 out (782 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-m.1:low__json_schema PASS 100% 149 in → 1,339 out (1,315 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
poolside/laguna-m.1:medium PASS 100% 151 in → 580 out (563 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-m.1:medium__json_schema PASS 100% 151 in → 428 out (403 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
poolside/laguna-m.1:none PASS 100% 151 in → 22 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-m.1:xhigh PASS 100% 151 in → 420 out (402 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-m.1:xhigh__json_schema PASS 100% 149 in → 1,557 out (1,529 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
poolside/laguna-xs-2.1:high PASS 100% 147 in → 404 out (389 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-xs-2.1:high__json_schema PASS 100% 151 in → 296 out (268 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
poolside/laguna-xs-2.1:low PASS 100% 151 in → 589 out (573 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-xs-2.1:low__json_schema PASS 100% 149 in → 343 out (315 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
poolside/laguna-xs-2.1:medium PASS 100% 150 in → 384 out (368 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 151 in → 3,052 out (3,026 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
poolside/laguna-xs-2.1:none PASS 100% 150 in → 15 out (0 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-xs-2.1:none__json_schema PASS 100% 150 in → 28 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
poolside/laguna-xs-2.1:xhigh PASS 100% 151 in → 2,244 out (2,230 reasoning)
def greet(name): print("Hi there") return False
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 149 in → 436 out (407 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
poolside/laguna-xs.2:high PASS 100% 186 in → 183 out (160 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-xs.2:high__json_schema PASS 100% 185 in → 265 out (239 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
poolside/laguna-xs.2:low PASS 100% 188 in → 243 out (220 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-xs.2:low__json_schema PASS 100% 186 in → 186 out (156 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
poolside/laguna-xs.2:medium PASS 100% 184 in → 238 out (215 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-xs.2:medium__json_schema PASS 100% 187 in → 167 out (141 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
poolside/laguna-xs.2:none PASS 100% 188 in → 22 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-xs.2:none__json_schema PASS 100% 186 in → 28 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
poolside/laguna-xs.2:xhigh PASS 100% 186 in → 262 out (239 reasoning)
```python def greet(name): print("Hi there") return False ```
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 186 in → 259 out (233 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
qwen/qwen3.6-27b:high PASS 100% 148 in → 1,147 out (1,194 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-27b:high__json_schema PASS 100% 147 in → 1,056 out (1,083 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-27b:low PASS 100% 147 in → 1,188 out (1,227 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-27b:low__json_schema PASS 100% 146 in → 1,292 out (1,346 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-27b:medium PASS 100% 148 in → 2,902 out (3,074 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.6-27b:medium__json_schema PASS 100% 147 in → 879 out (903 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-27b:none PASS 100% 148 in → 21 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.6-27b:xhigh PASS 100% 146 in → 1,442 out (1,495 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 147 in → 1,197 out (1,244 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-35b-a3b:high PASS 100% 148 in → 1,237 out (1,309 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 147 in → 869 out (915 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-35b-a3b:low PASS 100% 145 in → 1,049 out (1,066 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 147 in → 1,526 out (1,534 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 148 in → 1,828 out (1,926 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 147 in → 928 out (962 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-35b-a3b:none PASS 100% 150 in → 16 out (0 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 148 in → 1,020 out (1,087 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 148 in → 1,338 out (1,374 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
qwen/qwen3.6-flash:high PASS 100% 144 in → 1,045 out (1,025 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-flash:low PASS 100% 147 in → 1,200 out (1,180 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-flash:medium PASS 100% 147 in → 1,612 out (1,587 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.6-flash:none PASS 100% 149 in → 15 out (0 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.6-flash:xhigh PASS 100% 147 in → 1,485 out (1,465 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:high PASS 100% 148 in → 1,818 out (1,798 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:low PASS 100% 148 in → 1,695 out (1,675 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:low__json_schema PASS 100% 147 in → 1,491 out (1,471 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:max PASS 100% 148 in → 1,221 out (1,196 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.7-flash:medium PASS 100% 146 in → 1,512 out (1,492 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:none PASS 100% 149 in → 15 out (0 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-flash:xhigh PASS 100% 147 in → 1,160 out (1,140 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-max:high PASS 100% 146 in → 546 out (526 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-max:low PASS 100% 148 in → 699 out (679 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-max:medium PASS 100% 148 in → 741 out (721 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-max:none PASS 100% 150 in → 15 out (0 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-max:xhigh PASS 100% 148 in → 942 out (917 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.7-plus:high PASS 100% 147 in → 655 out (635 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-plus:low PASS 100% 147 in → 2,172 out (2,152 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-plus:medium PASS 100% 147 in → 1,144 out (1,124 reasoning)
def greet(name): print("Hi there") return False
qwen/qwen3.7-plus:none PASS 100% 150 in → 20 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
qwen/qwen3.7-plus:xhigh PASS 100% 146 in → 885 out (865 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:high PASS 100% 144 in → 1,052 out (1,037 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:high__json_schema PASS 100% 144 in → 552 out (556 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
tencent/hy3:low PASS 100% 144 in → 892 out (877 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:low__json_schema PASS 100% 146 in → 680 out (692 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
tencent/hy3:medium PASS 100% 146 in → 890 out (875 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:medium__json_schema PASS 100% 144 in → 819 out (837 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
tencent/hy3:none PASS 100% 146 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:none__json_schema PASS 100% 149 in → 29 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
tencent/hy3:xhigh PASS 100% 147 in → 1,452 out (1,436 reasoning)
def greet(name): print("Hi there") return False
tencent/hy3:xhigh__json_schema PASS 100% 144 in → 2,703 out (2,950 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling-small:high PASS 100% 144 in → 687 out (782 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling-small:high__json_schema PASS 100% 146 in → 1,266 out (1,234 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling-small:low PASS 100% 144 in → 109 out (106 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling-small:low__json_schema PASS 100% 145 in → 149 out (116 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling-small:max PASS 100% 144 in → 2,662 out (3,049 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling-small:max__json_schema PASS 100% 144 in → 723 out (691 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling-small:medium PASS 100% 143 in → 199 out (213 reasoning)
--- greet.py def greet(name): print("Hi there") return False
thinkingmachines/inkling-small:medium__json_schema PASS 100% 143 in → 168 out (136 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 143 in → 1,469 out (1,692 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 145 in → 1,119 out (1,085 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling:high PASS 100% 145 in → 3,739 out (3,723 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling:high__json_schema PASS 100% 144 in → 194 out (162 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling:low PASS 100% 143 in → 61 out (41 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling:low__json_schema PASS 100% 145 in → 51 out (19 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling:max PASS 100% 144 in → 80 out (60 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling:medium PASS 100% 144 in → 134 out (115 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling:none__json_schema PASS 100% 143 in → 28 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
thinkingmachines/inkling:xhigh PASS 100% 145 in → 326 out (369 reasoning)
def greet(name): print("Hi there") return False
thinkingmachines/inkling:xhigh__json_schema PASS 100% 142 in → 1,115 out (1,083 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
x-ai/grok-4.3:high PASS 100% 312 in → 612 out (599 reasoning)
def greet(name): print("Hi there") return False
x-ai/grok-4.3:high__json_schema PASS 100% 371 in → 586 out (562 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
x-ai/grok-4.3:low PASS 100% 318 in → 431 out (418 reasoning)
def greet(name): print("Hi there") return False
x-ai/grok-4.3:low__json_schema PASS 100% 375 in → 434 out (409 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
x-ai/grok-4.3:max PASS 100% 311 in → 364 out (346 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.3:max__json_schema PASS 100% 371 in → 508 out (488 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content":"def greet(name):\n print(\"Hi there\")\n return False"}
x-ai/grok-4.3:medium PASS 100% 318 in → 472 out (459 reasoning)
def greet(name): print("Hi there") return False
x-ai/grok-4.3:medium__json_schema PASS 100% 375 in → 567 out (543 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
x-ai/grok-4.3:none PASS 100% 311 in → 18 out (0 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.3:none__json_schema PASS 100% 369 in → 24 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
x-ai/grok-4.3:xhigh PASS 100% 312 in → 606 out (588 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.3:xhigh__json_schema PASS 100% 369 in → 542 out (518 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
x-ai/grok-4.5:high PASS 100% 341 in → 649 out (628 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.5:high__json_schema PASS 100% 410 in → 558 out (529 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
x-ai/grok-4.5:low PASS 100% 340 in → 415 out (394 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.5:low__json_schema PASS 100% 410 in → 489 out (460 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
x-ai/grok-4.5:max PASS 100% 341 in → 523 out (502 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.5:max__json_schema PASS 100% 411 in → 300 out (274 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
x-ai/grok-4.5:medium PASS 100% 342 in → 596 out (575 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.5:medium__json_schema PASS 100% 411 in → 500 out (474 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
x-ai/grok-4.5:xhigh PASS 100% 343 in → 515 out (494 reasoning)
```python def greet(name): print("Hi there") return False ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 411 in → 532 out (506 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n"}
z-ai/glm-4.7-flash:high PASS 100% 134 in → 1,108 out (1,181 reasoning)
```python def greet(name): print("Hi there") return False ```
z-ai/glm-4.7-flash:low PASS 100% 134 in → 478 out (484 reasoning)
```python def greet(name): print("Hi there") return False ```
z-ai/glm-4.7-flash:low__json_schema PASS 100% 136 in → 407 out (384 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
z-ai/glm-4.7-flash:medium PASS 100% 135 in → 406 out (399 reasoning)
```python def greet(name): print("Hi there") return False ```
z-ai/glm-4.7-flash:none PASS 100% 134 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
z-ai/glm-4.7-flash:none__json_schema PASS 100% 134 in → 26 out (0 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
z-ai/glm-4.7-flash:xhigh PASS 100% 133 in → 461 out (478 reasoning)
```python def greet(name): print("Hi there") return False ```
z-ai/glm-5.2:high PASS 100% 135 in → 771 out (811 reasoning)
def greet(name): print("Hi there") return False
z-ai/glm-5.2:high__json_schema PASS 100% 141 in → 1,019 out (1,095 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
z-ai/glm-5.2:low PASS 100% 141 in → 514 out (549 reasoning)
def greet(name): print("Hi there") return False
z-ai/glm-5.2:low__json_schema PASS 100% 142 in → 306 out (336 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
z-ai/glm-5.2:medium PASS 100% 141 in → 553 out (538 reasoning)
def greet(name): print("Hi there") return False
z-ai/glm-5.2:medium__json_schema PASS 100% 142 in → 531 out (503 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False\n" }
z-ai/glm-5.2:none PASS 100% 135 in → 14 out (0 reasoning)
def greet(name): print("Hi there") return False
z-ai/glm-5.2:xhigh PASS 100% 143 in → 252 out (276 reasoning)
```python def greet(name): print("Hi there") return False ```
z-ai/glm-5.2:xhigh__json_schema PASS 100% 144 in → 497 out (470 reasoning)
def greet(name): print("Hi there") return False
JSON Schema Structured Output
(raw) { "resolved_content": "def greet(name):\n print(\"Hi there\")\n return False" }
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:medium FAIL 0% 137 in → 741 out (844 reasoning)
def greet(name): print("Hi there") return False
Failure: Content mismatch: ['greet.py']
deepseek/deepseek-v4-flash:xhigh FAIL 0% 220 in → 913 out (1,062 reasoning)
defgreet(name): print("Hi there") return False
Failure: Content mismatch: ['greet.py']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 149 in → 706 out (685 reasoning)
def greet(name): print("Hi there"); return False
JSON Schema Structured Output
(raw) {"resolved_content": "def greet(name): print(\"Hi there\"); return False"}
Failure: Content mismatch: ['greet.py']
Invalid JSON. Output: ```python def greet(name): print("Hi there") return False ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: ``` def greet(name): print("Hi there") return False ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high FAIL 0% 307 in → 208 out (210 reasoning)
```pythondef greet(name): print("Hi there") return False ```
Failure: Content mismatch: ['greet.py']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/devstral-2512 FAIL 0% 144 in → 26 out
Here's the resolved file content: ```python def greet(name): print("Hi there") return False ```
Failure: Content mismatch: ['greet.py']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:high FAIL 0% 139 in → 617 out (721 reasoning)
def greet(name): print("Hi there") return False
Failure: Content mismatch: ['greet.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output: def greet(name): print("Hi there") return False
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:high FAIL 0% 138 in → 126 out (111 reasoning)
def greet(name): print("Hi there") return False
Failure: Content mismatch: ['greet.py']
moonshotai/kimi-k2.7-code:low FAIL 0% 138 in → 115 out (100 reasoning)
def greet(name): print("Hi there") return False
Failure: Content mismatch: ['greet.py']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-753fe816-c774-93c4-92ec-b1b9f049785b","request_id":"753fe816-c774-93c4-92ec-b1b9f049785b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43fbf649-350f-930a-b9d4-34f1a0fcd764","request_id":"43fbf649-350f-930a-b9d4-34f1a0fcd764"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0c620304-186b-9971-a5b9-ab07339e540f","request_id":"0c620304-186b-9971-a5b9-ab07339e540f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-43ce0fce-24d8-9477-ad60-95f43e13ee9d","request_id":"43ce0fce-24d8-9477-ad60-95f43e13ee9d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a457a970-34ec-9138-b6dc-77ed3a062423","request_id":"a457a970-34ec-9138-b6dc-77ed3a062423"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-cf69dfb2-2d3c-94e1-9120-e40d86812cf4"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-681dc584-f999-9c0a-928f-6d4220d195fe"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a62b6588-2d6a-9769-87e5-b8058279986e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5b4527e2-499e-9a39-8574-324442d1c42f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-057eb4bf-d7b6-92cc-b9f8-a7213cbac375"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9c64df54-eb2d-90c9-8b4e-9265fc3be841","request_id":"9c64df54-eb2d-90c9-8b4e-9265fc3be841"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7943e1e0-a9b7-9cdc-89bb-f8ce768ffe99","request_id":"7943e1e0-a9b7-9cdc-89bb-f8ce768ffe99"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c13b6b31-efaf-9371-a81c-9da562ca0aa9","request_id":"c13b6b31-efaf-9371-a81c-9da562ca0aa9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-aede9158-f3ab-9c41-90aa-f9bb8ecc978a","request_id":"aede9158-f3ab-9c41-90aa-f9bb8ecc978a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0ac5e993-64a5-9a22-aeff-a00adf4132a4","request_id":"0ac5e993-64a5-9a22-aeff-a00adf4132a4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-310e3bd5-ffaf-9c48-8a40-ec6ec2a6e231","request_id":"310e3bd5-ffaf-9c48-8a40-ec6ec2a6e231"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-97a4c523-866c-9738-b3d6-ea3f6b837a13","request_id":"97a4c523-866c-9738-b3d6-ea3f6b837a13"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1e91e7dc-105b-9ca3-9281-9a79723a7956","request_id":"1e91e7dc-105b-9ca3-9281-9a79723a7956"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 143 in → 16 out (0 reasoning)
def greet(name): print("Hi there") return False<|end_message|>
Failure: Content mismatch: ['greet.py']
Invalid JSON. Output: def greet(name): print("Hi there") return False{"resolved_content": "def greet(name):\n print(\"Hi there\")\n return False"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: def greet(name): print("Hi there") return False
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: def greet(name): print("Hi there") return False
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 145 in → 16 out (0 reasoning)
def greet(name): print("Hi there") return False<|end_message|>
Failure: Content mismatch: ['greet.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.