Multi-file: conflicts in both main.py and utils.py
Tests ability to resolve simultaneous conflicts in two files with different resolution strategies per file. Evaluates multi-file conflict juggling.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'def main(): print("Running")' > main.py
  5. 05 printf 'DEBUG = True PORT = 8000' > utils.py
  6. 06 git add main.py utils.py
  7. 07 git commit -m 'Initial code'
  8. 08 git checkout -b feature
  9. 09 printf 'def main(): print("Running v2")' > main.py
  10. 10 printf 'DEBUG = False PORT = 8000' > utils.py
  11. 11 git add main.py utils.py
  12. 12 git commit -m 'Feature: update main and disable debug'
  13. 13 git checkout main
  14. 14 printf 'def main(): print("Running enterprise")' > main.py
  15. 15 printf 'DEBUG = True PORT = 9000' > utils.py
  16. 16 git add main.py utils.py
  17. 17 git commit -m 'Enterprise: update main and port'
  18. 18 git cherry-pick $(git rev-parse feature)
Prompt
Resolve the cherry-pick conflicts in main.py and utils.py. For main.py: the incoming commit says 'Running v2', the current branch (main) says 'Running enterprise'. For utils.py: the incoming commit set DEBUG=False with PORT=8000, the current branch kept DEBUG=True with PORT=9000. Resolve both conflicts preferring the more descriptive version for main.py and the safer DEBUG=True for utils.py. Provide ONLY the two resolved file contents, separated by a blank line, with each file prefixed by its filename.
Expected
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000

Scoped model quality, cost, API time, and token usage for cherry_pick/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 357 in → 143 out (65 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:low PASS 100% 355 in → 51 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:medium PASS 100% 353 in → 95 out (45 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:xhigh PASS 100% 353 in → 169 out (89 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:high PASS 100% 265 in → 716 out (585 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:low PASS 100% 267 in → 530 out (485 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:medium PASS 100% 269 in → 522 out (517 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:xhigh PASS 100% 269 in → 1,095 out (692 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:max PASS 100% 362 in → 587 out (69 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:none PASS 100% 362 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:xhigh PASS 100% 362 in → 192 out (79 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:high PASS 100% 355 in → 177 out (45 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:low PASS 100% 351 in → 147 out (59 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:max PASS 100% 357 in → 1,150 out (137 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:medium PASS 100% 355 in → 125 out (50 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:none PASS 100% 357 in → 51 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:xhigh PASS 100% 351 in → 321 out (64 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-4.6:medium PASS 100% 242 in → 163 out (110 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-5:high PASS 100% 353 in → 263 out (84 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:low PASS 100% 355 in → 140 out (64 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:medium PASS 100% 359 in → 219 out (67 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:none PASS 100% 355 in → 41 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:xhigh PASS 100% 355 in → 608 out (142 reasoning)
main.py: ```python def main(): print("Running enterprise") ``` utils.py: ```python DEBUG = True PORT = 9000 ```
arcee-ai/trinity-large-thinking:high PASS 100% 203 in → 337 out (311 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:low PASS 100% 205 in → 261 out (236 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:medium PASS 100% 207 in → 237 out (211 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:xhigh PASS 100% 203 in → 475 out (449 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high PASS 100% 203 in → 333 out (353 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high__json_schema PASS 100% 207 in → 239 out (232 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:low__json_schema PASS 100% 203 in → 313 out (312 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py":"def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:medium__json_schema PASS 100% 203 in → 235 out (227 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 209 in → 246 out (250 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
deepseek/deepseek-v4-flash-0731:high PASS 100% 215 in → 195 out (190 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 211 in → 377 out (377 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:low PASS 100% 209 in → 358 out (368 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 209 in → 700 out (859 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:max PASS 100% 292 in → 306 out (321 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 290 in → 527 out (546 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 209 in → 310 out (283 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 209 in → 608 out (616 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:none PASS 100% 209 in → 26 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 288 in → 667 out (642 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 209 in → 589 out (613 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:high PASS 100% 213 in → 209 out (183 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:low PASS 100% 209 in → 456 out (430 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:medium PASS 100% 209 in → 261 out (270 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 211 in → 275 out (226 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 211 in → 43 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:xhigh PASS 100% 290 in → 472 out (447 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 288 in → 515 out (538 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-pro:high PASS 100% 211 in → 394 out (368 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 413 in → 695 out (659 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
deepseek/deepseek-v4-pro:low PASS 100% 213 in → 500 out (475 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:medium PASS 100% 211 in → 380 out (354 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 413 in → 494 out (447 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-pro:none PASS 100% 213 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:xhigh PASS 100% 290 in → 1,250 out (1,224 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:high PASS 100% 231 in → 641 out (611 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:high__json_schema PASS 100% 309 in → 2,555 out (2,518 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:low PASS 100% 231 in → 560 out (530 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:low__json_schema PASS 100% 226 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3-flash-preview:medium PASS 100% 227 in → 950 out (920 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:medium__json_schema PASS 100% 311 in → 1,333 out (1,292 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:xhigh PASS 100% 227 in → 844 out (814 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 224 in → 2,875 out (2,838 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-flash-lite-preview:high PASS 100% 230 in → 1,156 out (1,126 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 228 in → 1,001 out (947 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.1-flash-lite-preview:medium PASS 100% 231 in → 599 out (569 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 230 in → 1,304 out (1,274 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 228 in → 1,498 out (1,444 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.1-pro-preview:high PASS 100% 227 in → 1,427 out (1,397 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 311 in → 1,461 out (1,424 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-pro-preview:low PASS 100% 231 in → 581 out (541 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
google/gemini-3.1-pro-preview:medium PASS 100% 227 in → 619 out (589 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:xhigh PASS 100% 229 in → 1,457 out (1,425 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 311 in → 2,059 out (2,022 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:high PASS 100% 232 in → 1,691 out (1,661 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 313 in → 1,811 out (1,770 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:max PASS 100% 228 in → 1,465 out (1,435 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 309 in → 1,815 out (1,774 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:medium PASS 100% 232 in → 969 out (939 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 230 in → 822 out (764 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.5-flash-lite:xhigh PASS 100% 225 in → 1,135 out (1,105 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:high PASS 100% 231 in → 1,451 out (1,421 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:high__json_schema PASS 100% 311 in → 1,874 out (1,829 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:low PASS 100% 227 in → 428 out (398 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:low__json_schema PASS 100% 311 in → 334 out (289 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:medium PASS 100% 231 in → 1,046 out (1,016 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:medium__json_schema PASS 100% 313 in → 949 out (904 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:xhigh PASS 100% 227 in → 1,534 out (1,504 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 313 in → 1,551 out (1,506 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:high PASS 100% 227 in → 1,681 out (1,651 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:high__json_schema PASS 100% 313 in → 2,746 out (2,705 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:low PASS 100% 230 in → 338 out (308 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:max PASS 100% 232 in → 1,130 out (1,100 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:max__json_schema PASS 100% 309 in → 1,270 out (1,225 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:medium PASS 100% 232 in → 1,391 out (1,361 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:medium__json_schema PASS 100% 232 in → 1,159 out (1,122 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.6-flash:xhigh PASS 100% 229 in → 1,344 out (1,314 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 230 in → 1,366 out (1,325 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemma-4-26b-a4b-it:high PASS 100% 242 in → 936 out (864 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 241 in → 2,452 out (2,270 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:low PASS 100% 246 in → 1,612 out (1,499 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 243 in → 1,584 out (1,452 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:medium PASS 100% 247 in → 3,532 out (3,283 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 247 in → 1,453 out (1,322 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 244 in → 51 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 241 in → 1,648 out (1,538 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 247 in → 1,024 out (884 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:high PASS 100% 245 in → 350 out (272 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:high__json_schema PASS 100% 240 in → 619 out (533 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemma-4-31b-it:low PASS 100% 246 in → 402 out (331 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:low__json_schema PASS 100% 246 in → 422 out (341 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:medium PASS 100% 250 in → 509 out (478 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:medium__json_schema PASS 100% 247 in → 651 out (559 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:none PASS 100% 238 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:none__json_schema PASS 100% 238 in → 50 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" }}
google/gemma-4-31b-it:xhigh PASS 100% 244 in → 435 out (377 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
ibm-granite/granite-4.1-8b PASS 100% 224 in → 26 out
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:high PASS 100% 238 in → 215 out (189 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:medium PASS 100% 236 in → 485 out (455 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:xhigh PASS 100% 238 in → 1,426 out (1,697 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 261 in → 1,015 out (1,012 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:high PASS 100% 236 in → 285 out (309 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:high__json_schema PASS 100% 387 in → 1,035 out (985 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:low PASS 100% 236 in → 661 out (775 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:low__json_schema PASS 100% 389 in → 484 out (434 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:medium PASS 100% 239 in → 1,060 out (1,035 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:medium__json_schema PASS 100% 385 in → 1,939 out (1,901 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
minimax/minimax-m2.7:xhigh PASS 100% 243 in → 749 out (724 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 387 in → 1,666 out (1,616 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:high PASS 100% 378 in → 419 out (447 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:high__json_schema PASS 100% 365 in → 42 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
minimax/minimax-m3:low PASS 100% 374 in → 168 out (154 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:low__json_schema PASS 100% 324 in → 42 out (40 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
minimax/minimax-m3:medium PASS 100% 378 in → 218 out (202 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:medium__json_schema PASS 100% 365 in → 42 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
minimax/minimax-m3:xhigh PASS 100% 374 in → 185 out (190 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:xhigh__json_schema PASS 100% 326 in → 50 out (48 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
mistralai/devstral-2512 PASS 100% 228 in → 29 out
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
mistralai/devstral-2512__json_schema PASS 100% 226 in → 40 out
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:high PASS 100% 240 in → 1,065 out (1,128 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 226 in → 4,521 out (4,908 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
mistralai/mistral-medium-3-5:low PASS 100% 236 in → 1,671 out (1,762 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 228 in → 3,115 out (3,123 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
mistralai/mistral-medium-3-5:medium PASS 100% 236 in → 738 out (744 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 228 in → 2,873 out (3,099 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
mistralai/mistral-medium-3-5:none PASS 100% 240 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:xhigh PASS 100% 236 in → 1,212 out (1,256 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 226 in → 2,026 out (2,089 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k2.6:high PASS 100% 207 in → 1,063 out (1,035 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:high__json_schema PASS 100% 205 in → 1,577 out (1,768 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
moonshotai/kimi-k2.6:low PASS 100% 205 in → 1,273 out (1,245 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:low__json_schema PASS 100% 206 in → 2,278 out (1,707 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.6:medium PASS 100% 205 in → 1,032 out (1,150 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 207 in → 2,380 out (2,329 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.6:xhigh PASS 100% 207 in → 1,873 out (1,847 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:high PASS 100% 209 in → 966 out (1,140 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 207 in → 788 out (894 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.7-code:low PASS 100% 205 in → 218 out (192 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 205 in → 519 out (559 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.7-code:medium PASS 100% 204 in → 578 out (552 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 205 in → 305 out (264 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
moonshotai/kimi-k2.7-code:xhigh PASS 100% 205 in → 405 out (377 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 299 in → 471 out (436 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k3:high PASS 100% 284 in → 427 out (388 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:high__json_schema PASS 100% 282 in → 434 out (395 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:low PASS 100% 282 in → 384 out (336 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
moonshotai/kimi-k3:low__json_schema PASS 100% 282 in → 601 out (562 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:max PASS 100% 286 in → 1,284 out (1,245 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:max__json_schema PASS 100% 286 in → 361 out (320 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium PASS 100% 284 in → 371 out (330 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium__json_schema PASS 100% 282 in → 503 out (464 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh PASS 100% 282 in → 507 out (468 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 284 in → 329 out (290 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 239 in → 792 out (873 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 241 in → 518 out (553 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 239 in → 687 out (719 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 237 in → 499 out (510 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 239 in → 29 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 239 in → 49 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 241 in → 405 out (448 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 239 in → 473 out (500 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 241 in → 219 out (184 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 241 in → 353 out (292 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 239 in → 260 out (271 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 239 in → 384 out (316 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 237 in → 417 out (446 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 239 in → 299 out (238 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 239 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 239 in → 51 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 239 in → 263 out (272 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 239 in → 289 out (232 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-5.4-mini:high PASS 100% 200 in → 215 out (185 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:low PASS 100% 202 in → 85 out (55 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:medium PASS 100% 204 in → 136 out (106 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:none PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:xhigh PASS 100% 204 in → 246 out (216 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-nano:xhigh PASS 100% 204 in → 586 out (556 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:high PASS 100% 202 in → 546 out (516 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:low PASS 100% 202 in → 51 out (21 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:medium PASS 100% 200 in → 224 out (194 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:none PASS 100% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:xhigh PASS 100% 204 in → 3,157 out (3,127 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:high PASS 100% 200 in → 308 out (278 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:low PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:medium PASS 100% 198 in → 328 out (298 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:none PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:xhigh PASS 100% 202 in → 526 out (496 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:high PASS 100% 202 in → 193 out (163 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:low PASS 100% 204 in → 159 out (129 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:max PASS 100% 202 in → 302 out (272 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:medium PASS 100% 204 in → 203 out (173 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:none PASS 100% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:xhigh PASS 100% 200 in → 222 out (192 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:high PASS 100% 202 in → 181 out (151 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:max PASS 100% 200 in → 245 out (215 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:medium PASS 100% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:none PASS 100% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:xhigh PASS 100% 200 in → 175 out (145 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:high PASS 100% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:low PASS 100% 206 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:max PASS 100% 202 in → 194 out (164 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:medium PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:none PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:xhigh PASS 100% 202 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high PASS 100% 265 in → 1,027 out (1,181 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high__json_schema PASS 100% 342 in → 1,822 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
openai/gpt-oss-120b:low PASS 100% 265 in → 370 out (384 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:low__json_schema PASS 100% 267 in → 162 out (144 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-120b:medium PASS 100% 265 in → 236 out (234 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:medium__json_schema PASS 100% 269 in → 398 out (349 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
openai/gpt-oss-120b:xhigh PASS 100% 261 in → 731 out (844 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 259 in → 519 out (546 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-20b:high PASS 100% 261 in → 526 out (607 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:high__json_schema PASS 100% 259 in → 870 out (811 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-oss-20b:low PASS 100% 263 in → 113 out (97 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:low__json_schema PASS 100% 259 in → 89 out (41 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "\ndef main():\n print(\"Running enterprise\")\n", "utils.py": "\nDEBUG = True\nPORT = 9000\n" } }
openai/gpt-oss-20b:medium PASS 100% 261 in → 130 out (105 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:medium__json_schema PASS 100% 263 in → 244 out (235 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"} }
openai/gpt-oss-20b:xhigh PASS 100% 261 in → 925 out (1,036 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 261 in → 727 out (808 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-m.1:high PASS 100% 225 in → 488 out (458 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:high__json_schema PASS 100% 227 in → 359 out (308 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
poolside/laguna-m.1:low PASS 100% 227 in → 747 out (716 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:low__json_schema PASS 100% 231 in → 1,234 out (1,182 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-m.1:medium PASS 100% 229 in → 445 out (414 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:medium__json_schema PASS 100% 227 in → 614 out (564 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-m.1:none__json_schema PASS 100% 227 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
poolside/laguna-m.1:xhigh PASS 100% 227 in → 299 out (266 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
poolside/laguna-m.1:xhigh__json_schema PASS 100% 231 in → 260 out (212 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:high PASS 100% 227 in → 973 out (942 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:high__json_schema PASS 100% 231 in → 989 out (940 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:low PASS 100% 231 in → 967 out (936 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:low__json_schema PASS 100% 231 in → 591 out (551 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs-2.1:medium PASS 100% 231 in → 488 out (457 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 229 in → 579 out (527 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
poolside/laguna-xs-2.1:none PASS 100% 227 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:none__json_schema PASS 100% 227 in → 38 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs-2.1:xhigh PASS 100% 229 in → 2,998 out (2,967 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:high PASS 100% 268 in → 210 out (179 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:high__json_schema PASS 100% 268 in → 556 out (513 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
poolside/laguna-xs.2:low PASS 100% 266 in → 246 out (215 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:low__json_schema PASS 100% 266 in → 185 out (144 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:none PASS 100% 266 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:none__json_schema PASS 100% 266 in → 38 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:xhigh PASS 100% 266 in → 186 out (155 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 266 in → 249 out (208 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-27b:high PASS 100% 226 in → 1,645 out (1,611 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:high__json_schema PASS 100% 226 in → 1,344 out (1,326 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-27b:low PASS 100% 222 in → 1,096 out (1,130 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:low__json_schema PASS 100% 224 in → 1,442 out (1,418 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-27b:medium PASS 100% 224 in → 1,601 out (1 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:medium__json_schema PASS 100% 226 in → 1,242 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-27b:none PASS 100% 228 in → 29 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:none__json_schema PASS 100% 228 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-27b:xhigh PASS 100% 222 in → 1,505 out (1,561 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:xhigh__json_schema PASS 100% 224 in → 1,583 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-35b-a3b:high PASS 100% 222 in → 2,238 out (2,231 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 224 in → 1,466 out (1,437 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:low PASS 100% 224 in → 1,539 out (1,488 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 226 in → 1,438 out (1,378 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:medium PASS 100% 224 in → 1,664 out (1,768 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 224 in → 1,644 out (1,577 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:none PASS 100% 226 in → 29 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:none__json_schema PASS 100% 222 in → 49 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 222 in → 1,578 out (1,555 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 224 in → 1,407 out (1,333 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-flash:high PASS 100% 224 in → 1,318 out (1,285 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:low PASS 100% 226 in → 1,574 out (1,541 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:medium PASS 100% 224 in → 1,879 out (1,846 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:none PASS 100% 226 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:xhigh PASS 100% 226 in → 1,328 out (1,295 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:high PASS 100% 224 in → 994 out (961 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:low PASS 100% 226 in → 1,653 out (1,620 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:max PASS 100% 226 in → 1,819 out (1,784 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:medium PASS 100% 222 in → 1,865 out (1,832 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:none PASS 100% 226 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:xhigh PASS 100% 224 in → 1,483 out (1,450 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:high PASS 100% 224 in → 1,865 out (1,832 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:low PASS 100% 224 in → 1,715 out (1,682 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:medium PASS 100% 224 in → 483 out (450 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:none PASS 100% 228 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:xhigh PASS 100% 222 in → 866 out (833 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:high PASS 100% 224 in → 1,408 out (1,375 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:medium PASS 100% 220 in → 415 out (380 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:none PASS 100% 228 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:high PASS 100% 212 in → 1,515 out (1,488 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:high__json_schema PASS 100% 216 in → 2,163 out (2,192 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:low PASS 100% 212 in → 888 out (862 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:low__json_schema PASS 100% 214 in → 474 out (434 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:medium PASS 100% 214 in → 605 out (578 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:medium__json_schema PASS 100% 210 in → 791 out (783 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:none PASS 100% 217 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:none__json_schema PASS 100% 219 in → 44 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:xhigh PASS 100% 214 in → 2,519 out (2,690 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:xhigh__json_schema PASS 100% 216 in → 1,180 out (1,247 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:high PASS 100% 209 in → 813 out (880 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:low PASS 100% 213 in → 140 out (127 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:low__json_schema PASS 100% 211 in → 274 out (226 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:max PASS 100% 211 in → 685 out (734 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:max__json_schema PASS 100% 209 in → 3,891 out (3,838 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:medium PASS 100% 209 in → 400 out (425 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:medium__json_schema PASS 100% 211 in → 403 out (356 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:xhigh PASS 100% 209 in → 1,871 out (2,119 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 209 in → 585 out (544 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
thinkingmachines/inkling:high PASS 100% 215 in → 677 out (647 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:max PASS 100% 211 in → 1,960 out (2,250 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:medium PASS 100% 209 in → 981 out (951 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:xhigh PASS 100% 209 in → 570 out (539 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:xhigh__json_schema PASS 100% 211 in → 996 out (947 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:high PASS 100% 384 in → 552 out (528 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:high__json_schema PASS 100% 444 in → 1,857 out (1,827 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:low PASS 100% 382 in → 220 out (196 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:low__json_schema PASS 100% 452 in → 375 out (333 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:max PASS 100% 380 in → 466 out (442 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:max__json_schema PASS 100% 446 in → 674 out (632 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:medium PASS 100% 386 in → 575 out (551 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:medium__json_schema PASS 100% 450 in → 502 out (460 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:none PASS 100% 378 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:none__json_schema PASS 100% 442 in → 34 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:xhigh PASS 100% 378 in → 356 out (332 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:xhigh__json_schema PASS 100% 440 in → 847 out (805 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.5:high PASS 100% 406 in → 810 out (784 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:high__json_schema PASS 100% 482 in → 640 out (591 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:low PASS 100% 408 in → 336 out (310 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:low__json_schema PASS 100% 482 in → 472 out (423 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:max PASS 100% 408 in → 582 out (556 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:max__json_schema PASS 100% 484 in → 548 out (499 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:medium PASS 100% 408 in → 488 out (462 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:medium__json_schema PASS 100% 488 in → 771 out (722 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:xhigh PASS 100% 408 in → 342 out (316 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:xhigh__json_schema PASS 100% 482 in → 556 out (517 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
z-ai/glm-4.7-flash:high PASS 100% 199 in → 1,099 out (1,225 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:high__json_schema PASS 100% 199 in → 832 out (829 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-4.7-flash:low PASS 100% 199 in → 839 out (870 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:medium PASS 100% 199 in → 1,425 out (1,585 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:high PASS 100% 208 in → 408 out (405 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:low PASS 100% 204 in → 763 out (737 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:low__json_schema PASS 100% 391 in → 1,114 out (1,066 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-5.2:medium PASS 100% 206 in → 247 out (219 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:xhigh PASS 100% 208 in → 212 out (227 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:xhigh__json_schema PASS 100% 206 in → 1,128 out (1,165 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
anthropic/claude-haiku-4.5:none FAIL 50% 242 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:high FAIL 50% 238 in → 460 out (201 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Reasoning:** - **main.py**: "Running enterprise" (HEAD) is more descriptive than the generic "Running v2", so the HEAD version is kept. - **utils.py**: `DEBUG = True` (HEAD) is the safer choice (debug mode on guards against silent failures in production debugging), and `PORT = 9000` is kept along with it as part of the current branch's configuration.
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:low FAIL 50% 234 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:medium FAIL 50% 240 in → 40 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:none FAIL 50% 240 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:xhigh FAIL 50% 240 in → 40 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.7:high FAIL 50% 356 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.7:low FAIL 50% 364 in → 45 out (0 reasoning)
main.py def main(): print("Running enterprise v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
anthropic/claude-opus-4.7:medium FAIL 50% 360 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:high FAIL 50% 236 in → 572 out (259 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py** → Kept `"Running enterprise"` (HEAD/main) over `"Running v2"` because it is more descriptive and meaningful than a bare version tag. - **utils.py** → Kept `DEBUG = True` (HEAD/main) as the safer choice (avoids accidentally disabling debug mode in the production branch). `PORT = 9000` is retained alongside it since both variables came from the same HEAD block and the port pairing is consistent with that environment's configuration.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:low FAIL 50% 238 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:max FAIL 50% 240 in → 810 out (254 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution notes:** - **main.py** → Kept `"Running enterprise"` (HEAD/main) over `"Running v2"` as it is the more descriptive, meaningful identifier for the running environment. - **utils.py** → Kept `DEBUG = True` and `PORT = 9000` (HEAD/main) entirely. `DEBUG = True` is the safer choice (retains visibility into errors/logs), and `PORT = 9000` is kept consistent with that same HEAD state since no instruction indicated splitting the two settings across versions.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:none FAIL 50% 236 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:xhigh FAIL 50% 242 in → 532 out (217 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py** → Chose `"Running enterprise"` (HEAD/main) over `"Running v2"` because it is more descriptive, conveying the deployment tier rather than a bare version tag. - **utils.py** → Chose `DEBUG = True` / `PORT = 9000` (HEAD/main) entirely. `DEBUG = True` is the safer choice as it preserves visibility into errors during runtime, and since both settings came from the same branch block, keeping `PORT = 9000` alongside it maintains internal consistency with the current branch's configuration.
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 50% 211 in → 49 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n" } }
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash:high__json_schema FAIL 50% 320 in → 41 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n"}}
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash:low__json_schema FAIL 50% 213 in → 695 out (647 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
google/gemini-3.1-flash-lite-preview:low FAIL 50% 229 in → 139 out (109 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 50% 313 in → 157 out (120 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 8000"}}
Failure: Content mismatch: ['utils.py']
google/gemini-3.5-flash-lite:low FAIL 50% 231 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
google/gemini-3.5-flash-lite:low__json_schema FAIL 50% 230 in → 41 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 8000"}}
Failure: Content mismatch: ['utils.py']
google/gemini-3.6-flash:low__json_schema FAIL 50% 313 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 8000" } }
Failure: Content mismatch: ['utils.py']
google/gemma-4-26b-a4b-it:none FAIL 50% 242 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9_000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9_000" } }
Failure: Content mismatch: ['utils.py']
inclusionai/ling-2.6-flash FAIL 50% 244 in → 36 out
main.py def main(): print("Running enterprise - v2 enterprise deployment") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
main.py: def main(): print("Running enterprise v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise v2\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
Failure: Content mismatch: ['main.py']
liquid/lfm-2-24b-a2b FAIL 50% 233 in → 27 out
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
minimax/minimax-m2.5:high__json_schema FAIL 50% 238 in → 766 out (834 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
Failure: Content mismatch: ['main.py']
minimax/minimax-m2.5:low__json_schema FAIL 50% 259 in → 1,048 out (1,001 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9_000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9_000\n" } }
Failure: Content mismatch: ['utils.py']
minimax/minimax-m2.5:medium__json_schema FAIL 50% 236 in → 601 out (608 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")\n\n", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
mistralai/mistral-medium-3-5:none__json_schema FAIL 50% 228 in → 41 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
Failure: Content mismatch: ['main.py']
nvidia/nemotron-3-nano-30b-a3b:high__json_schema FAIL 50% 241 in → 705 out (681 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
openai/gpt-5.4-nano:high FAIL 50% 204 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-5.4-nano:low FAIL 50% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-5.4-nano:none FAIL 50% 200 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
openai/gpt-5.6-sol:low FAIL 50% 204 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
poolside/laguna-m.1:none FAIL 50% 231 in → 31 out (0 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
poolside/laguna-xs-2.1:xhigh__json_schema FAIL 50% 231 in → 2,116 out (2,072 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n\n", "utils.py": "DEBUG = True\nPORT = 8000\n"}}
Failure: Content mismatch: ['utils.py']
poolside/laguna-xs.2:medium FAIL 50% 266 in → 255 out (222 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
poolside/laguna-xs.2:medium__json_schema FAIL 50% 264 in → 296 out (252 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n"}}
Failure: Content mismatch: ['utils.py']
qwen/qwen3.7-plus:xhigh FAIL 50% 226 in → 757 out (724 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
thinkingmachines/inkling-small:none FAIL 50% 209 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
thinkingmachines/inkling:low FAIL 50% 209 in → 193 out (186 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
thinkingmachines/inkling:low__json_schema FAIL 50% 211 in → 350 out (308 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
Failure: Content mismatch: ['main.py']
thinkingmachines/inkling:none__json_schema FAIL 50% 207 in → 73 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": {"main.py": "def main():\n print(\"Running v2\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"} }
Failure: Content mismatch: ['main.py']
z-ai/glm-4.7-flash:none FAIL 50% 201 in → 25 out (0 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
z-ai/glm-4.7-flash:none__json_schema FAIL 50% 201 in → 51 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
z-ai/glm-4.7-flash:xhigh FAIL 50% 199 in → 1,097 out (1,099 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
z-ai/glm-5.2:none FAIL 50% 200 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-fable-5:high__json_schema FAIL 0% 594 in → 120 out (87 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:low__json_schema FAIL 0% 592 in → 47 out (24 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:max FAIL 0% 355 in → 1,435 out (176 reasoning)
``` main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000 ```
Failure: Expected file heading before file content
anthropic/claude-fable-5:max__json_schema FAIL 0% 588 in → 1,282 out (193 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:medium__json_schema FAIL 0% 586 in → 53 out (45 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 588 in → 331 out (177 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 444 in → 3,187 out (1,250 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 446 in → 1,203 out (818 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 448 in → 1,997 out (1,038 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 416 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 444 in → 997 out (673 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 417 in → 365 out (195 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 419 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:max FAIL 0% 242 in → 545 out (218 reasoning)
**main.py** ```python def main(): print("Running enterprise") ``` **utils.py** ```python DEBUG = True PORT = 9000 ``` **Reasoning:** - **main.py**: `"Running enterprise"` (HEAD) is more descriptive than the generic `"Running v2"`, so HEAD wins. - **utils.py**: `DEBUG = True` (HEAD) is the safer option (debug mode on avoids silently hiding errors in production debugging). Since both settings belong to the same conflict block and the HEAD side carries the safer `DEBUG` value, `PORT = 9000` from HEAD is kept alongside it.
Failure: No file blocks found
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 419 in → 545 out (238 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 413 in → 60 out (30 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 413 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 415 in → 331 out (194 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 595 in → 16 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 595 in → 85 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 589 in → 720 out (81 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 591 in → 21 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 591 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 591 in → 21 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 590 in → 83 out (41 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 592 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 588 in → 387 out (108 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 590 in → 62 out (45 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 592 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 588 in → 233 out (63 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
Invalid JSON. Output: {"files":{
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 11 (char 10)
Failure: Failed to parse structured JSON response: Expecting property name enclosed in double quotes: line 1 column 11 (char 10)
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 419 in → 12 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 413 in → 1,082 out (259 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 413 in → 193 out (151 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 415 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 415 in → 836 out (207 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 588 in → 155 out (47 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 592 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 590 in → 100 out (49 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 592 in → 13 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 588 in → 230 out (94 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low FAIL 0% 205 in → 234 out (234 reasoning)
main.pydef main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
arcee-ai/trinity-mini:medium FAIL 0% 201 in → 296 out (329 reasoning)
print("Running enterprise") DEBUG = True PORT = 9000
Failure: No file blocks found
arcee-ai/trinity-mini:xhigh FAIL 0% 205 in → 274 out (288 reasoning)
main.pydef main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
deepseek/deepseek-v4-flash:none FAIL 0% 213 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise v2") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['main.py', 'utils.py']
Invalid JSON. Output: --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 211 in → 9 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "files": {} }
Failure: No file blocks found
deepseek/deepseek-v4-pro:xhigh__json_schema FAIL 0% 209 in → 50 out (0 reasoning)
main.py: def main(): print("Running enterprise v2") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise v2\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n" } }
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemini-3.1-flash-lite-preview:medium__json_schema FAIL 0% 230 in → 885 out (823 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")", "utils.py": "utils.py\nDEBUG = True\nPORT = 8000" } }
Failure: Duplicate file block: main.py
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 313 in → 647 out (615 reasoning)
main.py: def main(): print('Running enterprise') utils.py: DEBUG = True; PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main(): print('Running enterprise')","utils.py":"DEBUG = True; PORT = 9000"}}
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 313 in → 1,008 out (975 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True; PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main(): print(\"Running enterprise\")","utils.py":"DEBUG = True; PORT = 9000"}}
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemini-3.5-flash-lite:xhigh__json_schema FAIL 0% 226 in → 3,140 out (3,103 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True; PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main(): print(\"Running enterprise\")", "utils.py": "DEBUG = True; PORT = 9000"}}
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemma-4-31b-it:xhigh__json_schema FAIL 0% 248 in → 472 out (466 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
Invalid JSON. Output: main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m2.5:low FAIL 0% 245 in → 716 out (828 reasoning)
def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
moonshotai/kimi-k2.6:xhigh__json_schema FAIL 0% 209 in → 1,582 out (1,529 reasoning)
main.py: --- main.py def main(): print("Running enterprise") utils.py: --- utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "--- main.py\ndef main():\n print(\"Running enterprise\")\n", "utils.py": "--- utils.py\nDEBUG = True\nPORT = 9000\n" } }
Failure: Duplicate file block: main.py
nvidia/nemotron-3-nano-30b-a3b:high FAIL 0% 241 in → 32,768 out (878 reasoning)
(empty output)
Failure: No file blocks found
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
openai/gpt-5.4-nano:medium FAIL 0% 200 in → 39 out (0 reasoning)
main.py ```python def main(): print("Running v2") ``` utils.py ```python DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['main.py', 'utils.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-463d7c6b-517b-9daa-9de8-5174066eaa92","request_id":"463d7c6b-517b-9daa-9de8-5174066eaa92"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-bf8658cd-8c84-94f7-ba95-ac2439f27229","request_id":"bf8658cd-8c84-94f7-ba95-ac2439f27229"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-2bfa97dc-871b-9f5b-830b-37fa3529b397","request_id":"2bfa97dc-871b-9f5b-830b-37fa3529b397"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d0e6feb4-8c55-909f-b4fc-05b49855d0b1","request_id":"d0e6feb4-8c55-909f-b4fc-05b49855d0b1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-0b49207c-8e8c-9a13-b8b4-f8ebfc26ed73","request_id":"0b49207c-8e8c-9a13-b8b4-f8ebfc26ed73"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-7c10f7d1-a8cd-993f-9bef-681d2154aa14"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8617030b-808a-96d6-bcd8-359ad628053c"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-e446ed26-38c1-9ce7-83b2-18d596ef9390"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-9087cbe5-5505-9957-8e9f-37f4208f2cb0"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-a423b305-ed89-9d4f-8520-88f9ca3d810b"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c7ad363b-bb1f-9455-a309-b92626e130f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-abfdd693-75ee-9a05-9ac5-f997f6b9f9b0","request_id":"abfdd693-75ee-9a05-9ac5-f997f6b9f9b0"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ab2b153b-5ce0-98d7-8a6c-8ed4af4611fb","request_id":"ab2b153b-5ce0-98d7-8a6c-8ed4af4611fb"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-563aaa27-0ffe-97af-b100-8d986148dc19","request_id":"563aaa27-0ffe-97af-b100-8d986148dc19"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-7ed29d1b-d71e-9cf5-a7e1-4826f90272fd","request_id":"7ed29d1b-d71e-9cf5-a7e1-4826f90272fd"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-dc1e550a-b6a8-97c0-8f50-3bb8bfcb283c","request_id":"dc1e550a-b6a8-97c0-8f50-3bb8bfcb283c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3f84e657-4e79-9982-bcfb-89f5d767fa4c","request_id":"3f84e657-4e79-9982-bcfb-89f5d767fa4c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-plus:low FAIL 0% 224 in → 252 out (213 reasoning)
``` --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000 ```
Failure: Expected file heading before file content
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fdf42e20-5647-9ee9-81bc-6e9bc05eb158","request_id":"fdf42e20-5647-9ee9-81bc-6e9bc05eb158"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-17acdb7d-d88a-9f2a-ab99-ba6ad095af83","request_id":"17acdb7d-d88a-9f2a-ab99-ba6ad095af83"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c657beff-4fbd-9fae-9f8a-b33688429433","request_id":"c657beff-4fbd-9fae-9f8a-b33688429433"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a21a10fa-aafa-973d-b7fc-2c050bac446c","request_id":"a21a10fa-aafa-973d-b7fc-2c050bac446c"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:high__json_schema FAIL 0% 211 in → 7,646 out (7,593 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")", "utils.py": "utils.py\nDEBUG = True\nPORT = 9000" } }
Failure: Duplicate file block: main.py
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000{ "files": { "main.py": "def main():\\n print(\"Running enterprise\")\\n", "utils.py": "DEBUG = True\\nPORT = 9000\\n" } }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
thinkingmachines/inkling:none FAIL 0% 211 in → 28 out (0 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000<|end_message|>
Failure: Content mismatch: ['main.py', 'utils.py']
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: ACE{
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Model 'z-ai/glm-5.2:none' violated the reasoning_level=none invariant: provider reported 778 reasoning token(s)