f007 — rebase

← rebase
Three-way: conflicting port and protocol settings in same line range
Tests ability to resolve a three-way rebase conflict with port and protocol. Evaluates multi-dimensional conflict with rebase context.
hard rebase three-way port protocol conflict-resolution
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'PORT=8080 PROTOCOL=http' > config.txt
  5. 05 git add config.txt
  6. 06 git commit -m 'Initial config'
  7. 07 git checkout -b branch-a
  8. 08 printf 'PORT=8443 PROTOCOL=https' > config.txt
  9. 09 git add config.txt
  10. 10 git commit -m 'Use HTTPS port'
  11. 11 git checkout main
  12. 12 git checkout -b branch-b
  13. 13 printf 'PORT=9090 PROTOCOL=http' > config.txt
  14. 14 git add config.txt
  15. 15 git commit -m 'Change to non-standard port'
  16. 16 git checkout main
  17. 17 git merge branch-a
  18. 18 git rebase branch-b
Prompt
Resolve the rebase conflict in config.txt. Three different configurations were attempted: base PORT=8080/PROTOCOL=http, branch being rebased PORT=9090/PROTOCOL=http, main's version PORT=8443/PROTOCOL=https (from branch-a merge). Prefer the HTTPS configuration when there is a conflict. Provide ONLY the resolved config.txt content.
Expected
PORT=8443 PROTOCOL=https
Fixture Metrics

Scoped model quality, cost, API time, and token usage for rebase/f007.

Loading...
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (546)
anthropic/claude-fable-5:high PASS 100% 230 in → 52 out (23 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-fable-5:high__json_schema PASS 100% 450 in → 64 out (24 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-fable-5:low PASS 100% 229 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-fable-5:low__json_schema PASS 100% 450 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-fable-5:max PASS 100% 229 in → 710 out (86 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-fable-5:max__json_schema PASS 100% 449 in → 525 out (103 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-fable-5:medium PASS 100% 231 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-fable-5:medium__json_schema PASS 100% 450 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-fable-5:xhigh PASS 100% 229 in → 47 out (26 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-fable-5:xhigh__json_schema PASS 100% 449 in → 74 out (25 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-haiku-4.5:high PASS 100% 175 in → 345 out (276 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 345 in → 241 out (204 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-haiku-4.5:low PASS 100% 176 in → 246 out (221 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 344 in → 247 out (195 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-haiku-4.5:medium PASS 100% 174 in → 163 out (134 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 342 in → 319 out (283 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
anthropic/claude-haiku-4.5:none PASS 100% 149 in → 16 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 314 in → 27 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 176 in → 199 out (168 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 342 in → 300 out (264 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:high__json_schema PASS 100% 316 in → 118 out (72 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:low PASS 100% 145 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.6:low__json_schema PASS 100% 314 in → 26 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:max__json_schema PASS 100% 312 in → 623 out (372 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:medium PASS 100% 144 in → 17 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 313 in → 27 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:none PASS 100% 146 in → 16 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-opus-4.6:none__json_schema PASS 100% 312 in → 26 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 313 in → 119 out (74 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:high PASS 100% 234 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:high__json_schema PASS 100% 455 in → 34 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:low PASS 100% 235 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:low__json_schema PASS 100% 453 in → 34 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:max PASS 100% 234 in → 971 out (112 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:max__json_schema PASS 100% 457 in → 179 out (40 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:medium PASS 100% 236 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 455 in → 34 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:none PASS 100% 234 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:none__json_schema PASS 100% 455 in → 34 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.7:xhigh PASS 100% 234 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 457 in → 34 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:high PASS 100% 231 in → 63 out (27 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-opus-4.8:high__json_schema PASS 100% 451 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:low PASS 100% 230 in → 33 out (12 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.8:low__json_schema PASS 100% 451 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:max PASS 100% 231 in → 758 out (172 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.8:max__json_schema PASS 100% 450 in → 598 out (85 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:medium PASS 100% 231 in → 56 out (23 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-opus-4.8:medium__json_schema PASS 100% 449 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:none PASS 100% 230 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-opus-4.8:none__json_schema PASS 100% 450 in → 31 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-opus-4.8:xhigh PASS 100% 231 in → 91 out (38 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-opus-4.8:xhigh__json_schema PASS 100% 451 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-4.6:high PASS 100% 146 in → 148 out (115 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 314 in → 128 out (82 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-4.6:low PASS 100% 145 in → 16 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 315 in → 26 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-4.6:max PASS 100% 144 in → 158 out (122 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 314 in → 207 out (156 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-4.6:medium PASS 100% 146 in → 47 out (20 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 313 in → 26 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-4.6:none PASS 100% 149 in → 16 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:xhigh PASS 100% 144 in → 131 out (90 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 313 in → 129 out (83 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-5:high PASS 100% 229 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-5:high__json_schema PASS 100% 449 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-5:low PASS 100% 231 in → 18 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-sonnet-5:low__json_schema PASS 100% 449 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-5:medium PASS 100% 230 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-5:medium__json_schema PASS 100% 452 in → 32 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-5:none PASS 100% 230 in → 22 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
anthropic/claude-sonnet-5:none__json_schema PASS 100% 450 in → 31 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
anthropic/claude-sonnet-5:xhigh PASS 100% 231 in → 18 out (0 reasoning)
PORT=8443 PROTOCOL=https
anthropic/claude-sonnet-5:xhigh__json_schema PASS 100% 449 in → 66 out (24 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
arcee-ai/trinity-large-thinking:high PASS 100% 126 in → 858 out (861 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-large-thinking:low PASS 100% 124 in → 752 out (766 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-large-thinking:medium PASS 100% 125 in → 450 out (444 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-large-thinking:xhigh PASS 100% 125 in → 512 out (513 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-mini:high PASS 100% 125 in → 378 out (411 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-mini:high__json_schema PASS 100% 123 in → 356 out (383 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
arcee-ai/trinity-mini:low PASS 100% 125 in → 334 out (380 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-mini:low__json_schema PASS 100% 125 in → 424 out (458 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
arcee-ai/trinity-mini:medium PASS 100% 124 in → 373 out (415 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-mini:medium__json_schema PASS 100% 124 in → 377 out (432 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
arcee-ai/trinity-mini:xhigh PASS 100% 125 in → 330 out (361 reasoning)
PORT=8443 PROTOCOL=https
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 124 in → 260 out (290 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
deepseek/deepseek-v4-flash-0731:high PASS 100% 132 in → 134 out (123 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 210 in → 274 out (255 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 130 in → 205 out (195 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 131 in → 144 out (134 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 224 in → 316 out (306 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 210 in → 237 out (230 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 130 in → 249 out (246 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 130 in → 227 out (204 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 132 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 137 in → 19 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n"}
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 131 in → 202 out (195 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 130 in → 159 out (149 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash:high PASS 100% 132 in → 149 out (138 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 133 in → 156 out (134 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
deepseek/deepseek-v4-flash:low PASS 100% 130 in → 155 out (159 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 133 in → 210 out (187 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash:medium PASS 100% 131 in → 211 out (199 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 229 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-flash:none PASS 100% 133 in → 9 out (0 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:xhigh PASS 100% 210 in → 255 out (245 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 208 in → 222 out (204 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-pro:high PASS 100% 132 in → 523 out (512 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 323 in → 345 out (327 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
deepseek/deepseek-v4-pro:low PASS 100% 131 in → 279 out (267 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 130 in → 144 out (138 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-pro:medium PASS 100% 130 in → 239 out (228 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 133 in → 179 out (158 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
deepseek/deepseek-v4-pro:none PASS 100% 132 in → 13 out (0 reasoning)
--- config.txt PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-pro:xhigh PASS 100% 210 in → 669 out (658 reasoning)
PORT=8443 PROTOCOL=https
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 130 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
google/gemini-3-flash-preview:high PASS 100% 137 in → 242 out (232 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3-flash-preview:high__json_schema PASS 100% 197 in → 792 out (775 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3-flash-preview:low PASS 100% 138 in → 205 out (195 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3-flash-preview:low__json_schema PASS 100% 198 in → 520 out (503 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3-flash-preview:medium PASS 100% 137 in → 323 out (313 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3-flash-preview:medium__json_schema PASS 100% 197 in → 879 out (862 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3-flash-preview:xhigh PASS 100% 135 in → 283 out (273 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 138 in → 281 out (264 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 139 in → 432 out (422 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-flash-lite-preview:low PASS 100% 138 in → 148 out (138 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 139 in → 143 out (122 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 136 in → 156 out (146 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 138 in → 192 out (174 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 137 in → 486 out (476 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 197 in → 1,480 out (1,463 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.1-pro-preview:high PASS 100% 137 in → 644 out (634 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-pro-preview:low PASS 100% 138 in → 380 out (370 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 196 in → 740 out (724 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.1-pro-preview:medium PASS 100% 137 in → 439 out (429 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 196 in → 818 out (801 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.1-pro-preview:xhigh PASS 100% 137 in → 668 out (658 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 198 in → 1,664 out (1,647 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.5-flash-lite:high PASS 100% 137 in → 442 out (432 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 197 in → 554 out (537 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.5-flash-lite:low PASS 100% 137 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 139 in → 23 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemini-3.5-flash-lite:max PASS 100% 136 in → 515 out (505 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 196 in → 1,187 out (1,170 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
google/gemini-3.5-flash-lite:medium PASS 100% 139 in → 326 out (316 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 197 in → 612 out (594 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 137 in → 369 out (359 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 198 in → 446 out (423 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemini-3.5-flash:high PASS 100% 138 in → 705 out (695 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash:high__json_schema PASS 100% 197 in → 643 out (623 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
google/gemini-3.5-flash:low PASS 100% 137 in → 384 out (374 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash:low__json_schema PASS 100% 197 in → 158 out (135 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemini-3.5-flash:medium PASS 100% 138 in → 549 out (539 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash:medium__json_schema PASS 100% 198 in → 584 out (559 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
google/gemini-3.5-flash:xhigh PASS 100% 136 in → 803 out (793 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 196 in → 850 out (832 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
google/gemini-3.6-flash:high PASS 100% 139 in → 420 out (410 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.6-flash:high__json_schema PASS 100% 137 in → 670 out (651 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
google/gemini-3.6-flash:low PASS 100% 139 in → 265 out (255 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.6-flash:low__json_schema PASS 100% 138 in → 23 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemini-3.6-flash:max PASS 100% 138 in → 391 out (381 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.6-flash:max__json_schema PASS 100% 197 in → 1,018 out (999 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
google/gemini-3.6-flash:medium PASS 100% 139 in → 447 out (437 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.6-flash:medium__json_schema PASS 100% 138 in → 822 out (803 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
google/gemini-3.6-flash:xhigh PASS 100% 139 in → 341 out (331 reasoning)
PORT=8443 PROTOCOL=https
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 198 in → 808 out (783 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
google/gemma-4-26b-a4b-it:high PASS 100% 153 in → 334 out (273 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 150 in → 480 out (395 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-26b-a4b-it:low PASS 100% 154 in → 336 out (271 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 154 in → 293 out (199 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-26b-a4b-it:medium PASS 100% 151 in → 284 out (219 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 153 in → 298 out (233 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-26b-a4b-it:none PASS 100% 150 in → 11 out (0 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 151 in → 23 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 153 in → 830 out (709 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 152 in → 292 out (212 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-31b-it:high PASS 100% 151 in → 205 out (160 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-31b-it:high__json_schema PASS 100% 154 in → 276 out (190 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-31b-it:low PASS 100% 154 in → 202 out (150 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-31b-it:low__json_schema PASS 100% 153 in → 279 out (205 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-31b-it:medium PASS 100% 153 in → 248 out (193 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-31b-it:medium__json_schema PASS 100% 153 in → 293 out (214 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-31b-it:none PASS 100% 147 in → 11 out (0 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-31b-it:none__json_schema PASS 100% 150 in → 24 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
google/gemma-4-31b-it:xhigh PASS 100% 154 in → 264 out (182 reasoning)
PORT=8443 PROTOCOL=https
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 153 in → 273 out (220 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
ibm-granite/granite-4.1-8b PASS 100% 137 in → 9 out
PORT=8443 PROTOCOL=https
ibm-granite/granite-4.1-8b__json_schema PASS 100% 136 in → 18 out
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
inclusionai/ling-2.6-flash PASS 100% 155 in → 11 out
PORT=8443 PROTOCOL=https
inclusionai/ling-2.6-flash__json_schema PASS 100% 155 in → 22 out
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.5:high PASS 100% 156 in → 138 out (136 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.5:high__json_schema PASS 100% 156 in → 298 out (272 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.5:low PASS 100% 154 in → 129 out (132 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.5:low__json_schema PASS 100% 158 in → 243 out (250 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.5:medium PASS 100% 155 in → 406 out (437 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.5:medium__json_schema PASS 100% 157 in → 302 out (316 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.5:xhigh PASS 100% 155 in → 266 out (256 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 170 in → 206 out (184 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.7:high PASS 100% 158 in → 392 out (383 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.7:high__json_schema PASS 100% 291 in → 516 out (497 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.7:low PASS 100% 155 in → 269 out (285 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.7:medium PASS 100% 156 in → 316 out (306 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m2.7:medium__json_schema PASS 100% 288 in → 461 out (442 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
minimax/minimax-m2.7:xhigh PASS 100% 156 in → 245 out (234 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m3:high PASS 100% 243 in → 177 out (164 reasoning)
``` PORT=8443 PROTOCOL=https ```
minimax/minimax-m3:high__json_schema PASS 100% 280 in → 19 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
minimax/minimax-m3:low PASS 100% 293 in → 216 out (223 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m3:medium PASS 100% 293 in → 235 out (242 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m3:medium__json_schema PASS 100% 281 in → 15 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
minimax/minimax-m3:xhigh PASS 100% 293 in → 108 out (114 reasoning)
PORT=8443 PROTOCOL=https
minimax/minimax-m3:xhigh__json_schema PASS 100% 280 in → 15 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/devstral-2512__json_schema PASS 100% 142 in → 19 out
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/mistral-medium-3-5:high PASS 100% 156 in → 450 out (408 reasoning)
PORT=8443 PROTOCOL=https
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 141 in → 524 out (480 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/mistral-medium-3-5:low PASS 100% 155 in → 451 out (412 reasoning)
PORT=8443 PROTOCOL=https
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 143 in → 256 out (229 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/mistral-medium-3-5:medium PASS 100% 156 in → 402 out (377 reasoning)
PORT=8443 PROTOCOL=https
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 144 in → 425 out (370 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
mistralai/mistral-medium-3-5:none PASS 100% 156 in → 15 out (0 reasoning)
``` PORT=8443 PROTOCOL=https ```
mistralai/mistral-medium-3-5:xhigh PASS 100% 155 in → 466 out (431 reasoning)
PORT=8443 PROTOCOL=https
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 144 in → 240 out (201 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
moonshotai/kimi-k2.6:high PASS 100% 126 in → 3,449 out (3,439 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k2.6:high__json_schema PASS 100% 124 in → 553 out (509 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
moonshotai/kimi-k2.6:medium PASS 100% 125 in → 284 out (275 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 124 in → 1,393 out (1,319 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
moonshotai/kimi-k2.6:xhigh PASS 100% 124 in → 311 out (302 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 124 in → 1,096 out (1,078 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n"}
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 125 in → 155 out (133 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 125 in → 227 out (208 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 124 in → 1,081 out (381 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
moonshotai/kimi-k3:high PASS 100% 202 in → 614 out (591 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:high__json_schema PASS 100% 202 in → 204 out (181 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:low PASS 100% 201 in → 495 out (472 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:low__json_schema PASS 100% 201 in → 195 out (172 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:max PASS 100% 204 in → 661 out (635 reasoning)
``` PORT=8443 PROTOCOL=https ```
moonshotai/kimi-k3:max__json_schema PASS 100% 202 in → 201 out (178 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:medium PASS 100% 205 in → 381 out (358 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:medium__json_schema PASS 100% 201 in → 309 out (286 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:xhigh PASS 100% 201 in → 561 out (538 reasoning)
PORT=8443 PROTOCOL=https
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 202 in → 384 out (361 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 156 in → 187 out (154 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 156 in → 89 out (85 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 155 in → 241 out (232 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 155 in → 158 out (139 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 155 in → 157 out (143 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 155 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 156 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 157 in → 121 out (112 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 156 in → 202 out (166 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 154 in → 146 out (106 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 156 in → 134 out (103 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 157 in → 66 out (37 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 156 in → 165 out (151 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 155 in → 146 out (109 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 155 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 156 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 156 in → 185 out (151 reasoning)
PORT=8443 PROTOCOL=https
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 156 in → 181 out (137 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
openai/gpt-5.4-mini:high PASS 100% 122 in → 87 out (73 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-mini:high__json_schema PASS 100% 157 in → 99 out (76 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-mini:low PASS 100% 122 in → 33 out (19 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-mini:low__json_schema PASS 100% 156 in → 44 out (21 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-mini:medium PASS 100% 120 in → 64 out (50 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-mini:medium__json_schema PASS 100% 157 in → 82 out (59 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-mini:none PASS 100% 120 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-mini:none__json_schema PASS 100% 155 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-mini:xhigh PASS 100% 122 in → 174 out (160 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 157 in → 492 out (469 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-nano:high PASS 100% 122 in → 44 out (30 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-nano:high__json_schema PASS 100% 158 in → 53 out (29 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
openai/gpt-5.4-nano:low PASS 100% 122 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-nano:low__json_schema PASS 100% 159 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
openai/gpt-5.4-nano:medium PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-nano:medium__json_schema PASS 100% 156 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
openai/gpt-5.4-nano:none PASS 100% 120 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-nano:none__json_schema PASS 100% 158 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4-nano:xhigh PASS 100% 120 in → 126 out (112 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 157 in → 116 out (92 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
openai/gpt-5.4:high PASS 100% 121 in → 140 out (126 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4:high__json_schema PASS 100% 159 in → 168 out (145 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4:low PASS 100% 122 in → 32 out (18 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4:low__json_schema PASS 100% 156 in → 49 out (26 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4:medium PASS 100% 120 in → 79 out (65 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4:medium__json_schema PASS 100% 156 in → 49 out (26 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4:none PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4:none__json_schema PASS 100% 156 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.4:xhigh PASS 100% 122 in → 138 out (124 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.4:xhigh__json_schema PASS 100% 156 in → 469 out (446 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.5:high PASS 100% 121 in → 136 out (122 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.5:high__json_schema PASS 100% 157 in → 470 out (447 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.5:low PASS 100% 121 in → 56 out (42 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.5:low__json_schema PASS 100% 158 in → 43 out (20 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.5:medium PASS 100% 120 in → 109 out (95 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.5:medium__json_schema PASS 100% 157 in → 181 out (158 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.5:none PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.5:none__json_schema PASS 100% 158 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.5:xhigh PASS 100% 122 in → 142 out (128 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.5:xhigh__json_schema PASS 100% 158 in → 211 out (188 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:high PASS 100% 122 in → 34 out (20 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:high__json_schema PASS 100% 156 in → 49 out (26 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:low PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:low__json_schema PASS 100% 156 in → 36 out (13 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:max PASS 100% 122 in → 51 out (37 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:max__json_schema PASS 100% 158 in → 326 out (303 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:medium PASS 100% 122 in → 31 out (17 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:medium__json_schema PASS 100% 156 in → 42 out (19 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:none PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:none__json_schema PASS 100% 156 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-luna:xhigh PASS 100% 120 in → 46 out (32 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 158 in → 126 out (103 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:high PASS 100% 122 in → 27 out (13 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:high__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:low PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:low__json_schema PASS 100% 158 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:max PASS 100% 122 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:max__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:medium PASS 100% 120 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:medium__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:none PASS 100% 120 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:none__json_schema PASS 100% 156 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-sol:xhigh PASS 100% 120 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:high PASS 100% 121 in → 28 out (14 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:high__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:low PASS 100% 123 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:low__json_schema PASS 100% 157 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:max PASS 100% 122 in → 53 out (39 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:max__json_schema PASS 100% 157 in → 229 out (206 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:medium PASS 100% 121 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:medium__json_schema PASS 100% 156 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:none PASS 100% 119 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:none__json_schema PASS 100% 158 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-5.6-terra:xhigh PASS 100% 122 in → 67 out (53 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 158 in → 21 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-oss-120b:high PASS 100% 181 in → 244 out (255 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-120b:high__json_schema PASS 100% 182 in → 328 out (303 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
openai/gpt-oss-120b:low PASS 100% 182 in → 42 out (28 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-120b:low__json_schema PASS 100% 169 in → 56 out (36 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
openai/gpt-oss-120b:medium PASS 100% 183 in → 141 out (144 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-120b:medium__json_schema PASS 100% 183 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
openai/gpt-oss-120b:xhigh PASS 100% 182 in → 115 out (107 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 181 in → 352 out (327 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
openai/gpt-oss-20b:high PASS 100% 183 in → 533 out (518 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-20b:high__json_schema PASS 100% 182 in → 17 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https\n"}
openai/gpt-oss-20b:low PASS 100% 182 in → 40 out (24 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-20b:low__json_schema PASS 100% 169 in → 53 out (30 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
openai/gpt-oss-20b:medium PASS 100% 186 in → 113 out (106 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-20b:medium__json_schema PASS 100% 183 in → 154 out (135 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
openai/gpt-oss-20b:xhigh PASS 100% 188 in → 448 out (489 reasoning)
PORT=8443 PROTOCOL=https
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 168 in → 255 out (266 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
poolside/laguna-m.1:high PASS 100% 145 in → 358 out (345 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-m.1:high__json_schema PASS 100% 143 in → 486 out (464 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-m.1:low PASS 100% 143 in → 271 out (258 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-m.1:low__json_schema PASS 100% 145 in → 343 out (321 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-m.1:medium PASS 100% 144 in → 381 out (368 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-m.1:medium__json_schema PASS 100% 142 in → 475 out (452 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-m.1:none PASS 100% 144 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-m.1:xhigh PASS 100% 145 in → 438 out (425 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-m.1:xhigh__json_schema PASS 100% 144 in → 382 out (359 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
poolside/laguna-xs-2.1:high PASS 100% 145 in → 314 out (303 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs-2.1:high__json_schema PASS 100% 144 in → 459 out (436 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs-2.1:low PASS 100% 145 in → 377 out (366 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs-2.1:low__json_schema PASS 100% 143 in → 340 out (316 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs-2.1:medium PASS 100% 144 in → 248 out (237 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 145 in → 504 out (482 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs-2.1:none PASS 100% 143 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs-2.1:none__json_schema PASS 100% 143 in → 24 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
poolside/laguna-xs-2.1:xhigh PASS 100% 144 in → 318 out (307 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 145 in → 272 out (250 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs.2:high PASS 100% 180 in → 191 out (178 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:high__json_schema PASS 100% 181 in → 207 out (188 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
poolside/laguna-xs.2:low PASS 100% 181 in → 193 out (180 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:low__json_schema PASS 100% 182 in → 205 out (187 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
poolside/laguna-xs.2:medium PASS 100% 180 in → 212 out (199 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:medium__json_schema PASS 100% 181 in → 200 out (176 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
poolside/laguna-xs.2:none PASS 100% 182 in → 12 out (0 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:none__json_schema PASS 100% 179 in → 22 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
poolside/laguna-xs.2:xhigh PASS 100% 182 in → 237 out (225 reasoning)
PORT=8443 PROTOCOL=https
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 181 in → 209 out (190 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
qwen/qwen3.6-27b:high PASS 100% 139 in → 1,122 out (1,096 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-27b:high__json_schema PASS 100% 141 in → 996 out (933 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-27b:low PASS 100% 140 in → 977 out (901 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-27b:low__json_schema PASS 100% 141 in → 904 out (851 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-27b:medium PASS 100% 141 in → 919 out (904 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-27b:medium__json_schema PASS 100% 140 in → 892 out (1 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-27b:none PASS 100% 143 in → 11 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-27b:xhigh PASS 100% 140 in → 1,080 out (1 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 141 in → 890 out (830 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-35b-a3b:high PASS 100% 139 in → 876 out (854 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 141 in → 1,433 out (1,353 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-35b-a3b:low PASS 100% 141 in → 1,135 out (1,085 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 141 in → 1,279 out (1,222 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 141 in → 830 out (815 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 141 in → 908 out (894 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-35b-a3b:none PASS 100% 143 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 138 in → 1,022 out (991 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 141 in → 1,234 out (1,172 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
qwen/qwen3.6-flash:high PASS 100% 140 in → 1,103 out (1,088 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-flash:low PASS 100% 140 in → 1,084 out (1,069 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-flash:medium PASS 100% 141 in → 1,081 out (1,066 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-flash:none PASS 100% 140 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.6-flash:xhigh PASS 100% 140 in → 927 out (912 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:high PASS 100% 139 in → 854 out (839 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:low PASS 100% 138 in → 1,129 out (1,114 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:max PASS 100% 140 in → 1,051 out (1,036 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:medium PASS 100% 141 in → 606 out (591 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:none PASS 100% 141 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-flash:xhigh PASS 100% 140 in → 1,057 out (1,042 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-max:high PASS 100% 139 in → 404 out (389 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-max:low PASS 100% 140 in → 240 out (225 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-max:medium PASS 100% 140 in → 273 out (258 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-max:none PASS 100% 141 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-max:xhigh PASS 100% 140 in → 378 out (363 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-plus:high PASS 100% 141 in → 237 out (222 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-plus:low PASS 100% 140 in → 325 out (310 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-plus:medium PASS 100% 141 in → 123 out (104 reasoning)
``` PORT=8443 PROTOCOL=https ```
qwen/qwen3.7-plus:none PASS 100% 142 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
qwen/qwen3.7-plus:xhigh PASS 100% 140 in → 366 out (351 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:high PASS 100% 137 in → 433 out (422 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:high__json_schema PASS 100% 136 in → 500 out (475 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
tencent/hy3:low PASS 100% 137 in → 437 out (426 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:low__json_schema PASS 100% 138 in → 421 out (390 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
tencent/hy3:medium PASS 100% 137 in → 523 out (496 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:medium__json_schema PASS 100% 138 in → 480 out (445 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
tencent/hy3:none PASS 100% 141 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:none__json_schema PASS 100% 140 in → 20 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
tencent/hy3:xhigh PASS 100% 136 in → 539 out (528 reasoning)
PORT=8443 PROTOCOL=https
tencent/hy3:xhigh__json_schema PASS 100% 136 in → 532 out (518 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
thinkingmachines/inkling-small:high PASS 100% 129 in → 320 out (319 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling-small:high__json_schema PASS 100% 129 in → 263 out (242 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
thinkingmachines/inkling-small:low PASS 100% 130 in → 75 out (61 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling-small:low__json_schema PASS 100% 131 in → 42 out (20 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
thinkingmachines/inkling-small:max PASS 100% 130 in → 237 out (239 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling-small:max__json_schema PASS 100% 129 in → 223 out (202 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
thinkingmachines/inkling-small:medium PASS 100% 131 in → 203 out (189 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling-small:medium__json_schema PASS 100% 131 in → 109 out (86 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
thinkingmachines/inkling-small:xhigh PASS 100% 131 in → 836 out (867 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 131 in → 209 out (187 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
thinkingmachines/inkling:high PASS 100% 130 in → 145 out (131 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:low PASS 100% 131 in → 28 out (14 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:max PASS 100% 131 in → 189 out (176 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:max__json_schema PASS 100% 131 in → 213 out (191 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
thinkingmachines/inkling:medium PASS 100% 131 in → 53 out (38 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:medium__json_schema PASS 100% 129 in → 74 out (51 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
thinkingmachines/inkling:none PASS 100% 130 in → 11 out (0 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:xhigh PASS 100% 130 in → 94 out (80 reasoning)
PORT=8443 PROTOCOL=https
thinkingmachines/inkling:xhigh__json_schema PASS 100% 131 in → 104 out (81 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
x-ai/grok-4.3:high PASS 100% 299 in → 326 out (318 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:high__json_schema PASS 100% 358 in → 513 out (500 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
x-ai/grok-4.3:low PASS 100% 305 in → 297 out (289 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:low__json_schema PASS 100% 362 in → 250 out (232 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
x-ai/grok-4.3:max PASS 100% 300 in → 262 out (254 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:max__json_schema PASS 100% 357 in → 361 out (344 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
x-ai/grok-4.3:medium PASS 100% 307 in → 347 out (339 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:medium__json_schema PASS 100% 363 in → 979 out (962 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
x-ai/grok-4.3:none PASS 100% 296 in → 8 out (0 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:none__json_schema PASS 100% 355 in → 13 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTOCOL=https"}
x-ai/grok-4.3:xhigh PASS 100% 301 in → 335 out (327 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.3:xhigh__json_schema PASS 100% 356 in → 444 out (427 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
x-ai/grok-4.5:high PASS 100% 321 in → 632 out (624 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.5:high__json_schema PASS 100% 391 in → 404 out (385 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
x-ai/grok-4.5:low PASS 100% 323 in → 155 out (147 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.5:low__json_schema PASS 100% 391 in → 341 out (323 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
x-ai/grok-4.5:max PASS 100% 321 in → 418 out (410 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.5:max__json_schema PASS 100% 391 in → 451 out (430 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
x-ai/grok-4.5:medium PASS 100% 321 in → 491 out (483 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.5:medium__json_schema PASS 100% 392 in → 670 out (652 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\n"}
x-ai/grok-4.5:xhigh PASS 100% 320 in → 481 out (473 reasoning)
PORT=8443 PROTOCOL=https
x-ai/grok-4.5:xhigh__json_schema PASS 100% 392 in → 602 out (581 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
z-ai/glm-4.7-flash:high PASS 100% 127 in → 383 out (357 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-4.7-flash:high__json_schema PASS 100% 126 in → 378 out (356 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-4.7-flash:low PASS 100% 126 in → 313 out (292 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-4.7-flash:low__json_schema PASS 100% 125 in → 468 out (426 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-4.7-flash:medium PASS 100% 127 in → 322 out (297 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-4.7-flash:medium__json_schema PASS 100% 126 in → 392 out (363 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-4.7-flash:none PASS 100% 124 in → 9 out (0 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-4.7-flash:none__json_schema PASS 100% 126 in → 19 out (0 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-4.7-flash:xhigh PASS 100% 126 in → 413 out (388 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 124 in → 372 out (339 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-5.2:high PASS 100% 133 in → 536 out (515 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-5.2:high__json_schema PASS 100% 132 in → 135 out (115 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
z-ai/glm-5.2:low PASS 100% 132 in → 93 out (82 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-5.2:low__json_schema PASS 100% 307 in → 283 out (266 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https"}
z-ai/glm-5.2:medium PASS 100% 132 in → 417 out (406 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-5.2:medium__json_schema PASS 100% 134 in → 396 out (372 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https\n" }
z-ai/glm-5.2:none PASS 100% 126 in → 10 out (0 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-5.2:xhigh PASS 100% 132 in → 354 out (324 reasoning)
PORT=8443 PROTOCOL=https
z-ai/glm-5.2:xhigh__json_schema PASS 100% 125 in → 430 out (386 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\nPROTOCOL=https" }
anthropic/claude-opus-4.6:high FAIL 0% 145 in → 240 out (89 reasoning)
``` PORT=8443 PROTOCOL=https ``` **Reasoning:** During the rebase, the branch's changes (`PORT=9090/PROTOCOL=http`) conflict with main's already-merged branch-a changes (`PORT=8443/PROTOCOL=https`). Per the instruction to prefer the HTTPS configuration, we accept main's version (`HEAD` here refers to the commit being rebased *onto*, but in this rebase context, the `=======`/`>>>>>>> cfa1452` side carries the HTTPS config). We resolve by keeping `PORT=8443` and `PROTOCOL=https`.
Failure: Content mismatch: ['config.txt']
anthropic/claude-opus-4.6:max FAIL 0% 145 in → 651 out (285 reasoning)
``` PORT=8443 PROTOCOL=https ``` **Reasoning:** The instruction says to prefer the HTTPS configuration on conflict. Both `PORT` and `PROTOCOL` conflict, so both values are taken from the `2cea58a (Use HTTPS port)` side (`PORT=8443` + `PROTOCOL=https`), discarding the rebased branch's `PORT=9090` / `PROTOCOL=http`.
Failure: Content mismatch: ['config.txt']
anthropic/claude-opus-4.6:xhigh FAIL 0% 146 in → 454 out (261 reasoning)
``` PORT=8443 PROTOCOL=https ``` **Reasoning:** The instruction is to prefer the HTTPS configuration when there is a conflict. Both `PORT` and `PROTOCOL` are in conflict, so we take the HTTPS-related values (`PORT=8443` and `PROTOCOL=https`) from the `5a4170c (Use HTTPS port)` side, discarding the `HEAD` side (`PORT=9090`/`PROTOCOL=http`).
Failure: Content mismatch: ['config.txt']
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 313 in → 25 out (0 reasoning)
PORT=8443 PROTECL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443\nPROTECL=https\n"}
Failure: Content mismatch: ['config.txt']
arcee-ai/trinity-large-thinking:high__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
arcee-ai/trinity-large-thinking:medium__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-large-thinking:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
google/gemini-3.1-flash-lite-preview:high__json_schema FAIL 0% 138 in → 2,393 out (2,377 reasoning)
PORT=8443 PROTOCOL=https
JSON Schema Structured Output
(raw) {"resolved_content":"PORT=8443 PROTOCOL=https"}
Failure: Content mismatch: ['config.txt']
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 197 in → 939 out (914 reasoning)
PORT=8443 PROTOCOL=httpshttps
JSON Schema Structured Output
(raw) {"resolved_content": "PORT=8443\nPROTOCOL=https\u0068ttps"}
Failure: Content mismatch: ['config.txt']
liquid/lfm-2-24b-a2b FAIL 0% 151 in → 31 out
Here is the resolved `config.txt` with the preferred HTTPS configuration: ``` PORT=8443 PROTOCOL=https ```
Failure: Content mismatch: ['config.txt']
liquid/lfm-2-24b-a2b__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.7:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
minimax/minimax-m2.7:xhigh__json_schema FAIL 0%
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
mistralai/devstral-2512 FAIL 0% 144 in → 30 out
Here's the resolved `config.txt` content with the HTTPS configuration preferred: ``` PORT=8443 PROTOCOL=https ```
Failure: Content mismatch: ['config.txt']
mistralai/mistral-medium-3-5:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.6:low FAIL 0% 125 in → 1,079 out (1,175 reasoning)
PORT=8443 PROTOCOL=https
Failure: Content mismatch: ['config.txt']
moonshotai/kimi-k2.6:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.7-code:high FAIL 0% 124 in → 356 out (346 reasoning)
PORT=8443 PROTOCOL=https
Failure: Content mismatch: ['config.txt']
moonshotai/kimi-k2.7-code:low FAIL 0% 126 in → 158 out (156 reasoning)
PORT=8443 PROTOCOL=https
Failure: Content mismatch: ['config.txt']
moonshotai/kimi-k2.7-code:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
moonshotai/kimi-k2.7-code:medium FAIL 0% 125 in → 147 out (142 reasoning)
PORT=8443 PROTOCOL=https
Failure: Content mismatch: ['config.txt']
moonshotai/kimi-k2.7-code:xhigh FAIL 0% 125 in → 159 out (149 reasoning)
PORT=8443 PROTOCOL=https
Failure: Content mismatch: ['config.txt']
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 0% 155 in → 112 out (92 reasoning)
PORT=8443\nPROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\\nPROTOCOL=https" }
Failure: Content mismatch: ['config.txt']
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema FAIL 0% 157 in → 185 out (166 reasoning)
PORT=8443\nPROTOCOL=https
JSON Schema Structured Output
(raw) { "resolved_content": "PORT=8443\\nPROTOCOL=https" }
Failure: Content mismatch: ['config.txt']
poolside/laguna-m.1:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-27b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.6-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-33d45284-23c6-9338-83a3-e3adffe48eaa","request_id":"33d45284-23c6-9338-83a3-e3adffe48eaa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-36ea7c2e-d3c0-9925-ab5f-e6decc960af9","request_id":"36ea7c2e-d3c0-9925-ab5f-e6decc960af9"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-edd6ccb0-c397-9df3-8d29-bf5d45d259e3","request_id":"edd6ccb0-c397-9df3-8d29-bf5d45d259e3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6db4785b-4b72-9c66-b0b4-5e3556c9673e","request_id":"6db4785b-4b72-9c66-b0b4-5e3556c9673e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-74ec080c-b036-97a6-ae49-494909fed768","request_id":"74ec080c-b036-97a6-ae49-494909fed768"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-82fb828a-f6f3-9f0f-8e3a-a3ba456bf3a2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e0c6c0aa-2ebd-968b-b31b-86c00d24ac4e"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-cf691082-f8aa-993b-9b7e-e10f061e4b57"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-84361568-51ee-9ded-baa0-e2744d00ae77"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-0974f9e4-457c-9d37-8532-9d35b6f1d764"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-68c30e11-4651-9cfa-a1cb-21e142ee3159"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dc844905-588c-9fea-af9a-bfda5aeeb025","request_id":"dc844905-588c-9fea-af9a-bfda5aeeb025"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f80a7da8-d39f-9ce3-966e-aaea30f335b1","request_id":"f80a7da8-d39f-9ce3-966e-aaea30f335b1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e6917d58-b3b0-968c-b22c-f400b69844f0","request_id":"e6917d58-b3b0-968c-b22c-f400b69844f0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-max:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-max:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14306903-d158-92f8-a34d-b39194311659","request_id":"14306903-d158-92f8-a34d-b39194311659"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:high__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6bfdfa51-b4f8-9e45-aff6-267c0839628d","request_id":"6bfdfa51-b4f8-9e45-aff6-267c0839628d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f4bfc742-0914-9bbb-b92b-e6b1a1c1c48e","request_id":"f4bfc742-0914-9bbb-b92b-e6b1a1c1c48e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:medium__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-03f2d03a-4ec6-93ab-87c2-508598a4f511","request_id":"03f2d03a-4ec6-93ab-87c2-508598a4f511"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.
qwen/qwen3.7-plus:xhigh__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c610baaa-c9f5-9771-b783-b6402f7ac4fb","request_id":"c610baaa-c9f5-9771-b783-b6402f7ac4fb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 131 in → 11 out (0 reasoning)
PORT=8443 PROTOCOL=https<|end_message|>
Failure: Content mismatch: ['config.txt']
thinkingmachines/inkling-small:none__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https{"resolved_content": "PORT=8443\nPROTOCOL=https"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:high__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:low__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none__json_schema FAIL 0%
Invalid JSON. Output: PORT=8443 PROTOCOL=https{"resolved_content": "PORT=8443\nPROTOCOL=https"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:none__json_schema FAIL 0%
(empty output)
JSON Schema Structured Output
Failure: Connection error.