Multi-line config block conflict
Tests ability to resolve a config block conflict during rebase. Evaluates infrastructure-merge judgment with rebase polarity.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'server: port: 8080 host: localhost' > config.yaml
  5. 05 git add config.yaml
  6. 06 git commit -m 'Initial config'
  7. 07 git checkout -b prod
  8. 08 printf 'server: port: 443 host: localhost' > config.yaml
  9. 09 git add config.yaml
  10. 10 git commit -m 'Use HTTPS port'
  11. 11 git checkout main
  12. 12 printf 'server: port: 8080 host: 0.0.0.0' > config.yaml
  13. 13 git add config.yaml
  14. 14 git commit -m 'Listen on all interfaces'
  15. 15 git rebase prod
Prompt
Resolve the rebase conflict in config.yaml. The base had port: 8080 and host: localhost. Branch being rebased changed port to 443 (HTTPS) keeping localhost. Main branch kept port 8080 but changed host to 0.0.0.0. Provide ONLY the resolved file content.
Expected
server: port: 443 host: 0.0.0.0

Scoped model quality, cost, API time, and token usage for rebase/f006.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high__json_schema PASS 100% 420 in → 60 out (25 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:low PASS 100% 198 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:low__json_schema PASS 100% 420 in → 60 out (32 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:max PASS 100% 199 in → 946 out (164 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 420 in → 58 out (21 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 420 in → 176 out (45 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:high PASS 100% 175 in → 278 out (215 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 345 in → 401 out (305 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
anthropic/claude-haiku-4.5:low PASS 100% 176 in → 962 out (564 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 343 in → 380 out (307 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
anthropic/claude-haiku-4.5:medium PASS 100% 175 in → 2,068 out (932 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 342 in → 1,586 out (798 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:none PASS 100% 146 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 315 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 177 in → 459 out (378 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 344 in → 328 out (251 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
anthropic/claude-opus-4.6:high PASS 100% 147 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:high__json_schema PASS 100% 314 in → 141 out (75 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:low PASS 100% 148 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:low__json_schema PASS 100% 315 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 314 in → 573 out (325 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:medium PASS 100% 146 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 314 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:none PASS 100% 147 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 314 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.6:xhigh PASS 100% 148 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 315 in → 131 out (67 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:high PASS 100% 202 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:high__json_schema PASS 100% 425 in → 49 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:low PASS 100% 203 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:low__json_schema PASS 100% 425 in → 49 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:max PASS 100% 204 in → 1,788 out (353 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:max__json_schema PASS 100% 423 in → 1,414 out (320 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:medium PASS 100% 203 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 424 in → 49 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:none PASS 100% 204 in → 32 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:none__json_schema PASS 100% 425 in → 49 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 203 in → 858 out (214 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 424 in → 39 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:high__json_schema PASS 100% 417 in → 504 out (136 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:low PASS 100% 199 in → 436 out (144 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:low__json_schema PASS 100% 418 in → 256 out (86 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:max PASS 100% 198 in → 1,942 out (258 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:max__json_schema PASS 100% 420 in → 1,076 out (217 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 417 in → 372 out (137 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:none__json_schema PASS 100% 419 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 198 in → 820 out (241 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 420 in → 750 out (178 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 148 in → 206 out (128 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 316 in → 201 out (118 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 145 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 315 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 148 in → 932 out (299 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 313 in → 253 out (169 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 148 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 314 in → 38 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 146 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 316 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 315 in → 220 out (135 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:high PASS 100% 198 in → 133 out (56 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 418 in → 73 out (27 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:low PASS 100% 199 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:low__json_schema PASS 100% 418 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:medium PASS 100% 199 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 420 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:none PASS 100% 198 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 420 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 200 in → 142 out (81 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 420 in → 99 out (38 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 130 in → 1,386 out (1,374 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:low PASS 100% 132 in → 1,713 out (1,697 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:medium PASS 100% 130 in → 1,261 out (1,244 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-large-thinking:xhigh PASS 100% 131 in → 509 out (496 reasoning)
server: port: 443 host: 0.0.0.0
arcee-ai/trinity-mini:high__json_schema PASS 100% 132 in → 346 out (345 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
arcee-ai/trinity-mini:medium__json_schema PASS 100% 129 in → 656 out (637 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
arcee-ai/trinity-mini:xhigh PASS 100% 131 in → 545 out (542 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:high PASS 100% 136 in → 349 out (320 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 215 in → 154 out (111 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 136 in → 298 out (278 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 135 in → 394 out (327 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 232 in → 594 out (532 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 213 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 135 in → 206 out (173 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 141 in → 398 out (318 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
deepseek/deepseek-v4-flash-0731:none PASS 100% 134 in → 19 out (0 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 140 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 213 in → 2,191 out (2,171 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 213 in → 1,115 out (1,016 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-flash:high PASS 100% 135 in → 213 out (189 reasoning)
``` server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 231 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:low PASS 100% 135 in → 143 out (122 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 135 in → 242 out (193 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:medium PASS 100% 135 in → 361 out (340 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 230 in → 28 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
deepseek/deepseek-v4-flash:none PASS 100% 133 in → 25 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 137 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 214 in → 820 out (793 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 215 in → 3,110 out (3,145 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
deepseek/deepseek-v4-pro:high PASS 100% 135 in → 26 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 136 in → 550 out (517 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-pro:low PASS 100% 134 in → 828 out (807 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 136 in → 859 out (826 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
deepseek/deepseek-v4-pro:medium PASS 100% 135 in → 372 out (351 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 328 in → 361 out (331 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
deepseek/deepseek-v4-pro:none PASS 100% 134 in → 25 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
deepseek/deepseek-v4-pro:xhigh PASS 100% 213 in → 1,389 out (1,368 reasoning)
server: port: 443 host: 0.0.0.0
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 136 in → 626 out (594 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3-flash-preview:high PASS 100% 142 in → 927 out (905 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:high__json_schema PASS 100% 143 in → 574 out (544 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:low PASS 100% 141 in → 347 out (325 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:low__json_schema PASS 100% 201 in → 1,354 out (1,325 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:medium PASS 100% 140 in → 416 out (394 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:medium__json_schema PASS 100% 201 in → 1,063 out (1,033 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3-flash-preview:xhigh PASS 100% 140 in → 1,007 out (985 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 201 in → 1,197 out (1,168 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 143 in → 450 out (428 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 143 in → 892 out (862 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 141 in → 170 out (148 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 202 in → 157 out (122 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 142 in → 230 out (208 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 141 in → 525 out (503 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 199 in → 1,673 out (1,643 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:high PASS 100% 141 in → 1,119 out (1,097 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 202 in → 1,026 out (991 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.1-pro-preview:low PASS 100% 141 in → 540 out (518 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:medium PASS 100% 140 in → 655 out (633 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 202 in → 394 out (363 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 142 in → 903 out (881 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 202 in → 1,803 out (1,768 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:high PASS 100% 141 in → 639 out (617 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 142 in → 1,339 out (1,308 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash-lite:low PASS 100% 142 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 200 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash-lite:max PASS 100% 141 in → 546 out (524 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 143 in → 1,675 out (1,646 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:medium PASS 100% 140 in → 296 out (274 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 141 in → 692 out (663 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 142 in → 482 out (460 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 200 in → 979 out (950 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.5-flash:high__json_schema PASS 100% 200 in → 1,520 out (1,489 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash:low PASS 100% 141 in → 509 out (487 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:low__json_schema PASS 100% 201 in → 713 out (678 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.5-flash:medium PASS 100% 141 in → 922 out (900 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.5-flash:medium__json_schema PASS 100% 201 in → 698 out (666 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 201 in → 1,135 out (1,105 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemini-3.6-flash:high PASS 100% 141 in → 748 out (726 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:high__json_schema PASS 100% 142 in → 592 out (561 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:low PASS 100% 142 in → 254 out (232 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:low__json_schema PASS 100% 200 in → 35 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.6-flash:max PASS 100% 141 in → 494 out (472 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:max__json_schema PASS 100% 199 in → 518 out (483 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemini-3.6-flash:medium PASS 100% 141 in → 572 out (550 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:medium__json_schema PASS 100% 201 in → 617 out (586 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
google/gemini-3.6-flash:xhigh PASS 100% 142 in → 589 out (567 reasoning)
server: port: 443 host: 0.0.0.0
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 201 in → 1,129 out (1,100 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-26b-a4b-it:high PASS 100% 155 in → 1,737 out (1,403 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 156 in → 1,780 out (1,520 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:low PASS 100% 158 in → 1,466 out (1,172 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 155 in → 350 out (236 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:medium PASS 100% 158 in → 902 out (683 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 157 in → 573 out (430 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:none PASS 100% 154 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 155 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 158 in → 1,596 out (1,573 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 157 in → 634 out (457 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:high PASS 100% 158 in → 734 out (599 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:high__json_schema PASS 100% 157 in → 264 out (171 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-31b-it:low PASS 100% 157 in → 259 out (174 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:low__json_schema PASS 100% 160 in → 319 out (288 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-31b-it:medium PASS 100% 156 in → 234 out (146 reasoning)
server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:medium__json_schema PASS 100% 159 in → 280 out (249 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
google/gemma-4-31b-it:none PASS 100% 155 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
google/gemma-4-31b-it:none__json_schema PASS 100% 154 in → 37 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
google/gemma-4-31b-it:xhigh PASS 100% 157 in → 294 out (197 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 158 in → 261 out (167 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
ibm-granite/granite-4.1-8b PASS 100% 137 in → 20 out
server: port: 443 host: 0.0.0.0
inclusionai/ling-2.6-flash__json_schema PASS 100% 160 in → 29 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
minimax/minimax-m2.5:high PASS 100% 163 in → 236 out (195 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.5:high__json_schema PASS 100% 164 in → 225 out (181 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:low PASS 100% 162 in → 686 out (652 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:low__json_schema PASS 100% 172 in → 354 out (320 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
minimax/minimax-m2.5:medium PASS 100% 169 in → 365 out (339 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m2.5:medium__json_schema PASS 100% 161 in → 645 out (621 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.5:xhigh PASS 100% 164 in → 2,484 out (2,735 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 171 in → 644 out (614 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m2.7:high PASS 100% 160 in → 396 out (371 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:high__json_schema PASS 100% 296 in → 687 out (660 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
minimax/minimax-m2.7:low PASS 100% 164 in → 692 out (672 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:medium PASS 100% 161 in → 564 out (543 reasoning)
server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:medium__json_schema PASS 100% 295 in → 915 out (888 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
minimax/minimax-m2.7:xhigh PASS 100% 165 in → 416 out (392 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 296 in → 777 out (738 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:high PASS 100% 300 in → 302 out (264 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:high__json_schema PASS 100% 286 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:low PASS 100% 249 in → 266 out (240 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:medium PASS 100% 300 in → 534 out (475 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
minimax/minimax-m3:medium__json_schema PASS 100% 286 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
minimax/minimax-m3:xhigh PASS 100% 302 in → 218 out (181 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/devstral-2512 PASS 100% 145 in → 27 out
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/devstral-2512__json_schema PASS 100% 145 in → 30 out
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:high PASS 100% 157 in → 513 out (438 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:low PASS 100% 157 in → 438 out (352 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 144 in → 1,371 out (1,239 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
mistralai/mistral-medium-3-5:medium PASS 100% 156 in → 581 out (515 reasoning)
server: port: 443 host: 0.0.0.0
mistralai/mistral-medium-3-5:none PASS 100% 156 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
mistralai/mistral-medium-3-5:xhigh PASS 100% 157 in → 681 out (559 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:high PASS 100% 130 in → 1,668 out (1,647 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:high__json_schema PASS 100% 131 in → 30 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.6:low PASS 100% 130 in → 742 out (721 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 132 in → 1,321 out (1,140 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.6:xhigh PASS 100% 130 in → 1,611 out (1,622 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 131 in → 1,318 out (835 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.7-code:high PASS 100% 130 in → 235 out (214 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 132 in → 489 out (447 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
moonshotai/kimi-k2.7-code:low PASS 100% 131 in → 520 out (499 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.7-code:medium PASS 100% 131 in → 201 out (162 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 130 in → 432 out (400 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 131 in → 172 out (151 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 132 in → 285 out (254 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
moonshotai/kimi-k3:high PASS 100% 208 in → 913 out (874 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:high__json_schema PASS 100% 207 in → 579 out (540 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:low PASS 100% 209 in → 972 out (933 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:low__json_schema PASS 100% 208 in → 1,129 out (1,095 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:max PASS 100% 209 in → 749 out (710 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:max__json_schema PASS 100% 208 in → 875 out (836 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
moonshotai/kimi-k3:medium PASS 100% 208 in → 782 out (744 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:medium__json_schema PASS 100% 209 in → 956 out (922 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:xhigh PASS 100% 208 in → 158 out (124 reasoning)
server: port: 443 host: 0.0.0.0
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 306 in → 568 out (523 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 156 in → 798 out (712 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 158 in → 612 out (552 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 157 in → 316 out (247 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 158 in → 1,212 out (1,061 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 158 in → 896 out (769 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 158 in → 551 out (448 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 158 in → 1,172 out (1,000 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 156 in → 890 out (749 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 158 in → 434 out (363 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 155 in → 329 out (271 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 157 in → 471 out (405 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 158 in → 506 out (422 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 157 in → 350 out (296 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 157 in → 262 out (210 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 157 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 154 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 157 in → 197 out (154 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 158 in → 236 out (183 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-5.4-mini:high PASS 100% 127 in → 227 out (202 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:high__json_schema PASS 100% 161 in → 155 out (120 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:low PASS 100% 127 in → 66 out (41 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:low__json_schema PASS 100% 162 in → 114 out (78 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-mini:medium PASS 100% 126 in → 123 out (98 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:medium__json_schema PASS 100% 163 in → 182 out (147 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4-mini:none PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:none__json_schema PASS 100% 163 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-mini:xhigh PASS 100% 127 in → 541 out (516 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 163 in → 324 out (288 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:high PASS 100% 126 in → 277 out (252 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:none__json_schema PASS 100% 164 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4-nano:xhigh PASS 100% 127 in → 850 out (825 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 162 in → 727 out (691 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:high PASS 100% 128 in → 100 out (75 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:high__json_schema PASS 100% 164 in → 174 out (139 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:low PASS 100% 127 in → 53 out (28 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:low__json_schema PASS 100% 164 in → 62 out (27 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:medium PASS 100% 126 in → 64 out (39 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:medium__json_schema PASS 100% 164 in → 76 out (41 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.4:none PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:none__json_schema PASS 100% 164 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.4:xhigh PASS 100% 129 in → 185 out (160 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.4:xhigh__json_schema PASS 100% 163 in → 181 out (146 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:high PASS 100% 126 in → 266 out (241 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:high__json_schema PASS 100% 164 in → 511 out (475 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:low PASS 100% 128 in → 94 out (69 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:low__json_schema PASS 100% 164 in → 144 out (108 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.5:medium PASS 100% 125 in → 154 out (129 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:medium__json_schema PASS 100% 161 in → 207 out (172 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:none PASS 100% 129 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:none__json_schema PASS 100% 165 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.5:xhigh PASS 100% 127 in → 380 out (355 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.5:xhigh__json_schema PASS 100% 162 in → 552 out (516 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:high PASS 100% 128 in → 74 out (49 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:high__json_schema PASS 100% 163 in → 70 out (34 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:low PASS 100% 127 in → 69 out (44 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:low__json_schema PASS 100% 165 in → 67 out (31 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:max PASS 100% 125 in → 89 out (64 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:max__json_schema PASS 100% 164 in → 126 out (90 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:medium PASS 100% 128 in → 57 out (32 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:medium__json_schema PASS 100% 162 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:none PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:none__json_schema PASS 100% 162 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-luna:xhigh PASS 100% 129 in → 88 out (63 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 162 in → 99 out (63 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:high PASS 100% 127 in → 56 out (31 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:high__json_schema PASS 100% 163 in → 105 out (69 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:low PASS 100% 128 in → 60 out (35 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:low__json_schema PASS 100% 163 in → 63 out (28 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:max PASS 100% 125 in → 108 out (83 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:max__json_schema PASS 100% 163 in → 220 out (185 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:medium PASS 100% 128 in → 71 out (46 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:medium__json_schema PASS 100% 164 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-sol:none PASS 100% 128 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:none__json_schema PASS 100% 164 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-sol:xhigh PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 161 in → 164 out (128 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
openai/gpt-5.6-terra:high PASS 100% 126 in → 95 out (70 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:high__json_schema PASS 100% 163 in → 207 out (172 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:low PASS 100% 126 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:low__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:max PASS 100% 128 in → 126 out (101 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:max__json_schema PASS 100% 164 in → 158 out (123 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:medium PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:medium__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:none PASS 100% 128 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:none__json_schema PASS 100% 163 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-5.6-terra:xhigh PASS 100% 127 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 162 in → 136 out (101 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-oss-120b:high PASS 100% 185 in → 582 out (573 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
openai/gpt-oss-120b:high__json_schema PASS 100% 188 in → 753 out (721 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:low PASS 100% 187 in → 77 out (39 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:low__json_schema PASS 100% 184 in → 109 out (65 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:medium PASS 100% 175 in → 255 out (223 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:medium__json_schema PASS 100% 186 in → 180 out (140 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-120b:xhigh PASS 100% 174 in → 985 out (1,009 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 186 in → 470 out (477 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:high PASS 100% 185 in → 601 out (526 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:high__json_schema PASS 100% 185 in → 541 out (477 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:low PASS 100% 188 in → 310 out (291 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:low__json_schema PASS 100% 189 in → 128 out (87 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
openai/gpt-oss-20b:medium PASS 100% 187 in → 515 out (496 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:medium__json_schema PASS 100% 239 in → 442 out (368 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
openai/gpt-oss-20b:xhigh PASS 100% 190 in → 591 out (564 reasoning)
server: port: 443 host: 0.0.0.0
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 187 in → 671 out (606 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-m.1:high PASS 100% 148 in → 1,126 out (1,101 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:high__json_schema PASS 100% 152 in → 708 out (672 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-m.1:low PASS 100% 149 in → 1,481 out (1,456 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:low__json_schema PASS 100% 151 in → 963 out (928 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-m.1:medium PASS 100% 151 in → 509 out (484 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:medium__json_schema PASS 100% 152 in → 376 out (344 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-m.1:none PASS 100% 150 in → 29 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-m.1:xhigh PASS 100% 152 in → 726 out (697 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-m.1:xhigh__json_schema PASS 100% 151 in → 647 out (612 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:high PASS 100% 151 in → 1,121 out (1,096 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:high__json_schema PASS 100% 150 in → 511 out (476 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
poolside/laguna-xs-2.1:low PASS 100% 150 in → 1,095 out (1,068 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:low__json_schema PASS 100% 151 in → 514 out (478 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:medium PASS 100% 152 in → 515 out (488 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 152 in → 1,108 out (1,072 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:none__json_schema PASS 100% 148 in → 34 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
poolside/laguna-xs-2.1:xhigh PASS 100% 152 in → 808 out (785 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:high PASS 100% 189 in → 424 out (394 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:high__json_schema PASS 100% 189 in → 301 out (268 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-xs.2:low__json_schema PASS 100% 189 in → 346 out (314 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-xs.2:medium__json_schema PASS 100% 189 in → 543 out (508 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n"}
poolside/laguna-xs.2:none PASS 100% 189 in → 24 out (0 reasoning)
server: port: 443 host: 0.0.0.0
poolside/laguna-xs.2:none__json_schema PASS 100% 188 in → 29 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
poolside/laguna-xs.2:xhigh PASS 100% 189 in → 316 out (286 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 189 in → 301 out (265 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
qwen/qwen3.6-27b:high PASS 100% 148 in → 1,692 out (1,461 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:high__json_schema PASS 100% 148 in → 1,455 out (1 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
qwen/qwen3.6-27b:low PASS 100% 148 in → 1,856 out (1,654 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:low__json_schema PASS 100% 147 in → 1,762 out (1,615 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:medium PASS 100% 147 in → 2,069 out (2,042 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:medium__json_schema PASS 100% 147 in → 1,054 out (1 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-27b:none PASS 100% 147 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:xhigh PASS 100% 146 in → 1,900 out (1,668 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 147 in → 1,379 out (1 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:high PASS 100% 146 in → 1,382 out (1,355 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 146 in → 1,686 out (1,504 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:low PASS 100% 148 in → 1,598 out (1,380 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 148 in → 1,139 out (961 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 147 in → 1,184 out (1,157 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 147 in → 1,788 out (1,526 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-35b-a3b:none PASS 100% 149 in → 28 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 146 in → 1,364 out (1,208 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 146 in → 1,267 out (959 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
qwen/qwen3.6-flash:high PASS 100% 146 in → 1,905 out (1,878 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:low PASS 100% 147 in → 1,420 out (1,389 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:medium PASS 100% 147 in → 1,608 out (1,581 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.6-flash:xhigh PASS 100% 146 in → 1,273 out (1,242 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:high PASS 100% 147 in → 1,501 out (1,474 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:low PASS 100% 148 in → 1,251 out (1,224 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:max PASS 100% 147 in → 1,462 out (1,435 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:medium PASS 100% 148 in → 1,225 out (1,198 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-flash:none PASS 100% 149 in → 27 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-flash:xhigh PASS 100% 148 in → 1,380 out (1,353 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:high PASS 100% 148 in → 1,074 out (1,047 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:low PASS 100% 145 in → 353 out (321 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-max:medium PASS 100% 146 in → 1,218 out (1,191 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-max:none PASS 100% 149 in → 26 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:high PASS 100% 148 in → 231 out (199 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:medium PASS 100% 148 in → 206 out (174 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
qwen/qwen3.7-plus:none PASS 100% 150 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0
qwen/qwen3.7-plus:xhigh PASS 100% 148 in → 417 out (385 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
tencent/hy3:high PASS 100% 137 in → 2,482 out (2,461 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:high__json_schema PASS 100% 137 in → 3,932 out (3,823 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:low PASS 100% 137 in → 1,177 out (1,156 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:low__json_schema PASS 100% 137 in → 978 out (826 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:medium PASS 100% 136 in → 1,193 out (1,057 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:medium__json_schema PASS 100% 137 in → 1,393 out (1,224 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:none PASS 100% 141 in → 21 out (0 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:none__json_schema PASS 100% 142 in → 33 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
tencent/hy3:xhigh PASS 100% 136 in → 1,369 out (1,348 reasoning)
server: port: 443 host: 0.0.0.0
tencent/hy3:xhigh__json_schema PASS 100% 136 in → 2,158 out (2,079 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:high PASS 100% 136 in → 3,825 out (3,994 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:high__json_schema PASS 100% 136 in → 607 out (573 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling-small:low PASS 100% 136 in → 83 out (54 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:low__json_schema PASS 100% 136 in → 128 out (90 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:max PASS 100% 137 in → 5,395 out (5,370 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:max__json_schema PASS 100% 135 in → 1,822 out (1,786 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling-small:medium PASS 100% 137 in → 243 out (194 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:medium__json_schema PASS 100% 137 in → 304 out (266 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 135 in → 602 out (573 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 135 in → 2,753 out (2,719 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
thinkingmachines/inkling:high PASS 100% 135 in → 877 out (852 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:high__json_schema PASS 100% 135 in → 751 out (716 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling:low PASS 100% 135 in → 50 out (25 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:max PASS 100% 136 in → 2,235 out (2,210 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:medium PASS 100% 135 in → 316 out (290 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:medium__json_schema PASS 100% 137 in → 686 out (651 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
thinkingmachines/inkling:none PASS 100% 135 in → 23 out (0 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:none__json_schema PASS 100% 134 in → 32 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n"}
thinkingmachines/inkling:xhigh PASS 100% 135 in → 2,195 out (2,170 reasoning)
server: port: 443 host: 0.0.0.0
thinkingmachines/inkling:xhigh__json_schema PASS 100% 136 in → 733 out (698 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:high PASS 100% 306 in → 458 out (439 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:high__json_schema PASS 100% 362 in → 1,139 out (1,110 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.3:low PASS 100% 311 in → 526 out (507 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:low__json_schema PASS 100% 367 in → 410 out (384 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
x-ai/grok-4.3:max PASS 100% 305 in → 565 out (546 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:max__json_schema PASS 100% 362 in → 2,886 out (2,861 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.3:medium PASS 100% 311 in → 936 out (917 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:medium__json_schema PASS 100% 369 in → 1,239 out (1,209 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.3:none PASS 100% 305 in → 24 out (0 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.3:none__json_schema PASS 100% 360 in → 29 out (0 reasoning)
--- config.yaml server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"--- config.yaml\nserver:\n port: 443\n host: 0.0.0.0"}
x-ai/grok-4.3:xhigh PASS 100% 307 in → 1,046 out (1,027 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.3:xhigh__json_schema PASS 100% 362 in → 673 out (644 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
x-ai/grok-4.5:high PASS 100% 328 in → 445 out (421 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:high__json_schema PASS 100% 397 in → 653 out (620 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:low PASS 100% 327 in → 460 out (436 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:low__json_schema PASS 100% 395 in → 326 out (293 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:max PASS 100% 326 in → 1,041 out (1,022 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.5:max__json_schema PASS 100% 395 in → 862 out (829 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:medium PASS 100% 327 in → 828 out (809 reasoning)
server: port: 443 host: 0.0.0.0
x-ai/grok-4.5:medium__json_schema PASS 100% 395 in → 835 out (802 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
x-ai/grok-4.5:xhigh PASS 100% 328 in → 1,197 out (1,173 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
x-ai/grok-4.5:xhigh__json_schema PASS 100% 398 in → 769 out (736 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
z-ai/glm-4.7-flash:high PASS 100% 132 in → 1,862 out (1,744 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:low PASS 100% 132 in → 2,022 out (1,894 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:low__json_schema PASS 100% 131 in → 516 out (451 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:medium PASS 100% 133 in → 1,096 out (1,011 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:none__json_schema PASS 100% 132 in → 31 out (0 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-4.7-flash:xhigh PASS 100% 131 in → 3,698 out (3,300 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ```
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 131 in → 645 out (563 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:high PASS 100% 139 in → 358 out (336 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:high__json_schema PASS 100% 139 in → 577 out (544 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
z-ai/glm-5.2:low PASS 100% 139 in → 174 out (152 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:low__json_schema PASS 100% 137 in → 331 out (253 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0" }
z-ai/glm-5.2:medium PASS 100% 139 in → 159 out (132 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:medium__json_schema PASS 100% 138 in → 186 out (139 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 443\n host: 0.0.0.0\n" }
z-ai/glm-5.2:none PASS 100% 130 in → 21 out (0 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:xhigh PASS 100% 138 in → 312 out (291 reasoning)
server: port: 443 host: 0.0.0.0
z-ai/glm-5.2:xhigh__json_schema PASS 100% 314 in → 184 out (155 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 443\n host: 0.0.0.0\n"}
anthropic/claude-fable-5:high FAIL 0% 201 in → 128 out (28 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` This resolution combines both intended changes: the HTTPS port (443) from the branch being rebased and the all-interfaces binding (0.0.0.0) from main, since the two changes touch different settings and aren't mutually exclusive.
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
anthropic/claude-fable-5:medium FAIL 0% 199 in → 113 out (25 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` This merges both intents: the rebased branch's change to HTTPS (port 443) and main's change to listen on all interfaces (0.0.0.0), since each side modified a different line from the base.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-fable-5:xhigh FAIL 0% 199 in → 125 out (61 reasoning)
(empty output)
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.6:max FAIL 0% 146 in → 1,260 out (396 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** Each branch made a separate, non-overlapping change from the base: | Setting | Base | Rebased Branch (HEAD) | Main Branch | Resolution | |---|---|---|---|---| | `port` | 8080 | **443** *(changed)* | 8080 *(unchanged)* | **443** | | `host` | localhost | localhost *(unchanged)* | **0.0.0.0** *(changed)* | **0.0.0.0** | Both intentional changes are preserved: HTTPS (`443`) **and** listening on all interfaces (`0.0.0.0`).
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:high FAIL 0% 199 in → 853 out (127 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` Both changes are independent and should be preserved: the rebased branch's HTTPS port (443) and main's change to listen on all interfaces (0.0.0.0).
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:medium FAIL 0% 201 in → 569 out (139 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Reasoning:** Both branches made non-overlapping intentional changes that should be preserved: - The rebased branch's meaningful change was **port: 443** (switch to HTTPS). - Main's meaningful change was **host: 0.0.0.0** (listen on all interfaces). The correct merge keeps both, so the final config uses port 443 and host 0.0.0.0.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-opus-4.8:none FAIL 0% 199 in → 344 out (0 reasoning)
Looking at this rebase, I need to combine both intentional changes: - **Branch being rebased**: changed port to `443` (HTTPS) - **Main branch**: changed host to `0.0.0.0` (all interfaces) Both changes are complementary, so the resolution keeps both. **A note on the conflict markers during rebase:** During a rebase, `HEAD` refers to the branch you're rebasing *onto* (main, with `0.0.0.0`), and the incoming side (`c9fac88`) is your branch's commit (port `443`). So `HEAD` here is actually main's `port: 8080` / `host: 0.0.0.0`... but the labels in your paste show `HEAD` as `port: 443`, which is inverted from typical rebase behavior. Regardless of which side is which, the *intended* resolution combines both distinct changes: ```yaml server: port: 443 host: 0.0.0.0 ``` This keeps the HTTPS port from the rebased branch and the all-interfaces host from main.
Failure: Content mismatch: ['config.yaml']
anthropic/claude-sonnet-4.6:xhigh FAIL 0% 146 in → 902 out (255 reasoning)
```yaml server: port: 443 host: 0.0.0.0 ``` **Resolution logic:** | Field | Base | Branch (HEAD) | Main (01dc22d) | Resolution | |-------|------|---------------|----------------|------------| | `port` | 8080 | **443** (changed) | 8080 (unchanged) | **443** — accept the intentional HTTPS change | | `host` | localhost | localhost (unchanged) | **0.0.0.0** (changed) | **0.0.0.0** — accept the intentional interface binding change | Both changes are independent modifications to different fields. Each branch touched only one field, so both changes are taken — neither conflicts with the other's intent.
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:high FAIL 0% 130 in → 323 out (321 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:low FAIL 0% 130 in → 368 out (368 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:low__json_schema FAIL 0% 131 in → 604 out (648 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:medium FAIL 0% 130 in → 463 out (449 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
arcee-ai/trinity-mini:xhigh__json_schema FAIL 0% 130 in → 874 out (875 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: 0.0.0.0" }
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 200 in → 981 out (949 reasoning)
server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server: port: 443 host: 0.0.0.0" }
Failure: Content mismatch: ['config.yaml']
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 200 in → 170 out (136 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
google/gemini-3.5-flash:high FAIL 0% 142 in → 1,394 out (1,373 reasoning)
server port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
google/gemini-3.5-flash:xhigh FAIL 0% 140 in → 1,001 out (980 reasoning)
server port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
--- config.yaml server: port: 443 host: localhost ---
JSON Schema Structured Output
(raw) { "resolved_content": "--- config.yaml\nserver:\n port: 443\n host: localhost\n---\n" }
Failure: Content mismatch: ['config.yaml']
inclusionai/ling-2.6-flash FAIL 0% 161 in → 32 out
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
liquid/lfm-2-24b-a2b FAIL 0% 149 in → 27 out
```yaml server: port: 8080 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: ```yaml server: port: 443 host: localhost ```
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
minimax/minimax-m3:xhigh__json_schema FAIL 0% 286 in → 31 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "server:\n port: 8080\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
mistralai/mistral-medium-3-5:high__json_schema FAIL 0% 145 in → 3,608 out (3,439 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0"}
Failure: Content mismatch: ['config.yaml']
mistralai/mistral-medium-3-5:medium__json_schema FAIL 0% 145 in → 877 out (794 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0"}
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/mistral-medium-3-5:xhigh__json_schema FAIL 0% 143 in → 366 out (283 reasoning)
--- server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content": "---\nserver:\n port: 443\n host: 0.0.0.0"}
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:medium FAIL 0% 132 in → 3,075 out (3,054 reasoning)
server: port: 443 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
nvidia/nemotron-3-nano-30b-a3b:none FAIL 0% 157 in → 27 out (0 reasoning)
--- config.yaml server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
nvidia/nemotron-3-nano-30b-a3b:none__json_schema FAIL 0% 158 in → 35 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: 0.0.0.0\n" }
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:high__json_schema FAIL 0% 163 in → 35 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 8080\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:low FAIL 0% 126 in → 24 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:low__json_schema FAIL 0% 164 in → 35 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 8080\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:medium FAIL 0% 128 in → 24 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:medium__json_schema FAIL 0% 164 in → 35 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) {"resolved_content":"server:\n port: 8080\n host: 0.0.0.0\n"}
Failure: Content mismatch: ['config.yaml']
openai/gpt-5.4-nano:none FAIL 0% 126 in → 24 out (0 reasoning)
server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.
poolside/laguna-xs-2.1:none FAIL 0% 150 in → 36 out (0 reasoning)
server: port: 443 host: 0.0.0.0 >>>>>>> 7bbf7b7 (Listen on all interfaces)
Failure: Content mismatch: ['config.yaml']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 0% 152 in → 699 out (662 reasoning)
server: port: 8080 host: 0.0.0.0
JSON Schema Structured Output
(raw) { "resolved_content": "server:\n port: 8080\n host: 0.0.0.0\n" }
Failure: Content mismatch: ['config.yaml']
poolside/laguna-xs.2:low FAIL 0% 188 in → 379 out (346 reasoning)
``` --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
poolside/laguna-xs.2:medium FAIL 0% 187 in → 307 out (274 reasoning)
```yaml config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1fce3f3b-9f64-992b-b54e-ebc43a25f37c","request_id":"1fce3f3b-9f64-992b-b54e-ebc43a25f37c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f725b131-c086-93cb-944f-88c73cb50469","request_id":"f725b131-c086-93cb-944f-88c73cb50469"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-95ee5ee4-daa0-97c6-b5a9-a83bea0daa59","request_id":"95ee5ee4-daa0-97c6-b5a9-a83bea0daa59"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 149 in → 27 out (0 reasoning)
--- config.yaml server: port: 8080 host: 0.0.0.0
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-646a1955-3d5f-9e93-a7c3-9983b8ef5f3d","request_id":"646a1955-3d5f-9e93-a7c3-9983b8ef5f3d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3ee9a332-bae3-9471-9e03-3c0f24d221b0","request_id":"3ee9a332-bae3-9471-9e03-3c0f24d221b0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9fd30882-b221-97df-9638-8aece2a6c7f5"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-889044a7-3a04-9dad-b95a-91633abfbd00"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-83dc3f42-74f1-97b0-883d-41b80b205a51"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-633f4c09-4271-9237-a0d3-cfbbf55ef424"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-42dae19d-d18f-9c20-926b-d3f8d42a17dd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-3bc12337-1501-9089-86f9-f18459fa7d88"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f8293383-2144-9041-a278-cc72d8d31304","request_id":"f8293383-2144-9041-a278-cc72d8d31304"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a985deed-3fb9-90db-a317-fbf25ed89d1d","request_id":"a985deed-3fb9-90db-a317-fbf25ed89d1d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-242f4b94-c9ed-9f92-9e2f-8498d10b4ce2","request_id":"242f4b94-c9ed-9f92-9e2f-8498d10b4ce2"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-max:xhigh FAIL 0% 147 in → 746 out (710 reasoning)
```yaml --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c3fa5ff1-0ec3-920b-880c-e601bae8b9f6","request_id":"c3fa5ff1-0ec3-920b-880c-e601bae8b9f6"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b5e868ba-09e6-9a57-922d-30556fb8e920","request_id":"b5e868ba-09e6-9a57-922d-30556fb8e920"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low FAIL 0% 148 in → 1,509 out (1,474 reasoning)
``` --- config.yaml server: port: 443 host: 0.0.0.0 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e8bf57ef-8aba-97d3-9846-24c88b073f34","request_id":"e8bf57ef-8aba-97d3-9846-24c88b073f34"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-1bb47eda-7126-9f04-85ad-25026ea75ad0","request_id":"1bb47eda-7126-9f04-85ad-25026ea75ad0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Connection error.
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e50ee96e-b826-934d-863f-29a83067bbad","request_id":"e50ee96e-b826-934d-863f-29a83067bbad"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 137 in → 22 out (0 reasoning)
server: port: 443 host: 0.0.0.0<|end_message|>
Failure: Content mismatch: ['config.yaml']
Invalid JSON. Output: server: port: 443 host: 0.0.0.0{"resolved_content": "server:\n port: 443\n host: 0.0.0.0"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: server: port: 443 host: 0.0.0.0
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:none FAIL 0% 132 in → 26 out (0 reasoning)
```yaml server: port: 8080 host: 0.0.0.0 ```
Failure: Content mismatch: ['config.yaml']
(empty output)
JSON Schema Structured Output
Failure: Connection error.