Multi-file: conflicts in both main.py and utils.py
Tests ability to resolve simultaneous conflicts in two files with different strategies per file. Evaluates multi-file conflict juggling.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 printf 'def main(): print("Running")' > main.py
  5. 05 printf 'DEBUG = True PORT = 8000' > utils.py
  6. 06 git add main.py utils.py
  7. 07 git commit -m 'Initial code'
  8. 08 git checkout -b feature
  9. 09 printf 'def main(): print("Running v2")' > main.py
  10. 10 printf 'DEBUG = False PORT = 8000' > utils.py
  11. 11 git add main.py utils.py
  12. 12 git commit -m 'Feature: update main and disable debug'
  13. 13 git checkout main
  14. 14 printf 'def main(): print("Running enterprise")' > main.py
  15. 15 printf 'DEBUG = True PORT = 9000' > utils.py
  16. 16 git add main.py utils.py
  17. 17 git commit -m 'Enterprise: update main and port'
  18. 18 git merge feature
Prompt
Resolve the merge conflicts in main.py and utils.py. For main.py: branch A says 'Running v2', branch B says 'Running enterprise'. For utils.py: branch A set DEBUG=False with PORT=8000, branch B kept DEBUG=True with PORT=9000. Resolve both conflicts preferring the more descriptive version for main.py and the safer DEBUG=True for utils.py. Provide ONLY the two resolved file contents, separated by a blank line, with each file prefixed by its filename.
Expected
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000

Scoped model quality, cost, API time, and token usage for merge_conflicts/f010.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:high PASS 100% 306 in → 115 out (49 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:low PASS 100% 306 in → 99 out (36 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:max PASS 100% 306 in → 903 out (173 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-fable-5:medium PASS 100% 306 in → 119 out (58 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-fable-5:xhigh PASS 100% 306 in → 224 out (89 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:high PASS 100% 232 in → 214 out (169 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:low PASS 100% 232 in → 477 out (428 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:medium PASS 100% 232 in → 422 out (377 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-haiku-4.5:xhigh PASS 100% 232 in → 565 out (491 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:high PASS 100% 311 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:low PASS 100% 311 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:max PASS 100% 311 in → 569 out (114 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:medium PASS 100% 311 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:none PASS 100% 311 in → 43 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.7:xhigh PASS 100% 311 in → 177 out (51 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:high PASS 100% 306 in → 215 out (57 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:low PASS 100% 306 in → 91 out (36 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:max PASS 100% 306 in → 1,060 out (147 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
anthropic/claude-opus-4.8:medium PASS 100% 306 in → 155 out (60 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-opus-4.8:xhigh PASS 100% 306 in → 389 out (94 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-4.6:low PASS 100% 203 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-4.6:none PASS 100% 203 in → 41 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-5:high PASS 100% 306 in → 250 out (65 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:low PASS 100% 306 in → 106 out (43 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:medium PASS 100% 306 in → 126 out (54 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
anthropic/claude-sonnet-5:none PASS 100% 306 in → 51 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
anthropic/claude-sonnet-5:xhigh PASS 100% 306 in → 535 out (122 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:high PASS 100% 173 in → 1,465 out (1,447 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:low PASS 100% 173 in → 575 out (548 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:medium PASS 100% 173 in → 690 out (667 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-large-thinking:xhigh PASS 100% 173 in → 1,070 out (1,049 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high PASS 100% 173 in → 221 out (233 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
arcee-ai/trinity-mini:high__json_schema PASS 100% 173 in → 232 out (234 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
arcee-ai/trinity-mini:low__json_schema PASS 100% 173 in → 309 out (322 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:medium__json_schema PASS 100% 173 in → 242 out (244 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 173 in → 362 out (372 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
deepseek/deepseek-v4-flash-0731:high PASS 100% 257 in → 758 out (805 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 178 in → 1,217 out (1,247 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-flash-0731:low PASS 100% 178 in → 485 out (511 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 178 in → 499 out (494 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:max PASS 100% 257 in → 569 out (542 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 257 in → 1,338 out (1,390 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 184 in → 586 out (627 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 178 in → 830 out (867 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
deepseek/deepseek-v4-flash-0731:none PASS 100% 178 in → 27 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 257 in → 487 out (462 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:high PASS 100% 178 in → 320 out (295 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 258 in → 469 out (490 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
deepseek/deepseek-v4-flash:low PASS 100% 178 in → 337 out (328 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 178 in → 486 out (464 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-flash:medium PASS 100% 178 in → 574 out (548 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:xhigh PASS 100% 257 in → 502 out (476 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 257 in → 4,993 out (5,657 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-pro:high PASS 100% 178 in → 551 out (525 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:low PASS 100% 178 in → 573 out (548 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:medium PASS 100% 178 in → 861 out (835 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:none PASS 100% 178 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
deepseek/deepseek-v4-pro:none__json_schema PASS 100% 380 in → 46 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 458 in → 2,103 out (2,063 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:high PASS 100% 190 in → 801 out (771 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:high__json_schema PASS 100% 191 in → 959 out (918 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:low PASS 100% 190 in → 464 out (434 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:low__json_schema PASS 100% 272 in → 1,102 out (1,065 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:medium PASS 100% 190 in → 735 out (705 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:medium__json_schema PASS 100% 191 in → 1,321 out (1,280 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3-flash-preview:xhigh PASS 100% 190 in → 527 out (497 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 272 in → 2,067 out (2,030 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-flash-lite-preview:high PASS 100% 191 in → 1,343 out (1,313 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 191 in → 2,303 out (2,266 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-flash-lite-preview:low PASS 100% 190 in → 156 out (126 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:medium PASS 100% 190 in → 499 out (469 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 272 in → 531 out (481 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 191 in → 1,786 out (1,756 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 191 in → 2,501 out (2,464 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-pro-preview:high PASS 100% 190 in → 1,059 out (1,029 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:high__json_schema PASS 100% 272 in → 1,594 out (1,557 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-pro-preview:low PASS 100% 190 in → 560 out (530 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:medium PASS 100% 190 in → 847 out (817 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:medium__json_schema PASS 100% 272 in → 842 out (805 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.1-pro-preview:xhigh PASS 100% 190 in → 1,106 out (1,076 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 272 in → 2,438 out (2,397 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash-lite:high PASS 100% 191 in → 1,247 out (1,217 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 272 in → 1,460 out (1,419 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:low PASS 100% 191 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 272 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.5-flash-lite:max PASS 100% 191 in → 1,719 out (1,689 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 272 in → 1,375 out (1,334 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash-lite:medium PASS 100% 191 in → 999 out (969 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 272 in → 716 out (662 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemini-3.5-flash-lite:xhigh PASS 100% 191 in → 2,102 out (2,072 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 191 in → 1,178 out (1,137 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash:high PASS 100% 190 in → 1,284 out (1,254 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:high__json_schema PASS 100% 272 in → 1,598 out (1,553 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.5-flash:low PASS 100% 190 in → 409 out (377 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:low__json_schema PASS 100% 272 in → 476 out (418 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.5-flash:medium PASS 100% 190 in → 977 out (947 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:medium__json_schema PASS 100% 272 in → 633 out (592 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
google/gemini-3.5-flash:xhigh PASS 100% 190 in → 1,191 out (1,161 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 272 in → 1,950 out (1,892 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.6-flash:high PASS 100% 191 in → 1,097 out (1,067 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:high__json_schema PASS 100% 272 in → 1,103 out (1,045 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.6-flash:low PASS 100% 191 in → 494 out (464 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:low__json_schema PASS 100% 272 in → 58 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemini-3.6-flash:max PASS 100% 191 in → 1,211 out (1,181 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:max__json_schema PASS 100% 191 in → 1,497 out (1,456 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemini-3.6-flash:medium PASS 100% 191 in → 1,037 out (1,007 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:medium__json_schema PASS 100% 272 in → 1,233 out (1,196 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
google/gemini-3.6-flash:xhigh PASS 100% 191 in → 1,387 out (1,357 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 191 in → 1,934 out (1,893 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
google/gemma-4-26b-a4b-it:high PASS 100% 206 in → 1,037 out (942 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 206 in → 1,589 out (1,422 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:low PASS 100% 206 in → 2,717 out (2,664 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 205 in → 1,392 out (1,248 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:medium PASS 100% 205 in → 1,015 out (880 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 205 in → 1,716 out (1,552 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:none PASS 100% 203 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 203 in → 52 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 206 in → 1,136 out (1,006 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 205 in → 1,775 out (1,573 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
google/gemma-4-31b-it:high PASS 100% 206 in → 338 out (293 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:low PASS 100% 206 in → 362 out (285 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:low__json_schema PASS 100% 206 in → 304 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:medium PASS 100% 206 in → 315 out (249 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:medium__json_schema PASS 100% 206 in → 393 out (1 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
google/gemma-4-31b-it:none PASS 100% 203 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
google/gemma-4-31b-it:none__json_schema PASS 100% 203 in → 50 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" }}
google/gemma-4-31b-it:xhigh PASS 100% 206 in → 343 out (1 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
ibm-granite/granite-4.1-8b PASS 100% 189 in → 26 out
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:high PASS 100% 205 in → 471 out (506 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:high__json_schema PASS 100% 205 in → 401 out (421 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
minimax/minimax-m2.5:low PASS 100% 205 in → 687 out (666 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:low__json_schema PASS 100% 205 in → 205 out (187 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.5:medium PASS 100% 205 in → 911 out (1,024 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:medium__json_schema PASS 100% 205 in → 330 out (279 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.5:xhigh PASS 100% 205 in → 377 out (415 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 226 in → 770 out (725 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:high PASS 100% 207 in → 993 out (968 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:high__json_schema PASS 100% 204 in → 864 out (958 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
minimax/minimax-m2.7:low PASS 100% 204 in → 660 out (761 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:low__json_schema PASS 100% 354 in → 1,012 out (962 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:medium PASS 100% 207 in → 985 out (960 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:medium__json_schema PASS 100% 354 in → 1,052 out (1,002 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m2.7:xhigh PASS 100% 205 in → 636 out (713 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m2.7:xhigh__json_schema PASS 100% 354 in → 634 out (584 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:high PASS 100% 340 in → 263 out (265 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:high__json_schema PASS 100% 291 in → 51 out (49 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:low PASS 100% 343 in → 167 out (142 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:low__json_schema PASS 100% 330 in → 50 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
minimax/minimax-m3:medium PASS 100% 343 in → 329 out (354 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:medium__json_schema PASS 100% 330 in → 42 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
minimax/minimax-m3:xhigh PASS 100% 343 in → 218 out (208 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
minimax/minimax-m3:xhigh__json_schema PASS 100% 330 in → 43 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" }}
mistralai/devstral-2512__json_schema PASS 100% 189 in → 40 out
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:high PASS 100% 201 in → 1,368 out (1,419 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 189 in → 1,135 out (1,176 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:low PASS 100% 201 in → 1,556 out (1,617 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 189 in → 1,075 out (1,092 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:medium PASS 100% 201 in → 1,469 out (1,512 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 189 in → 2,608 out (2,797 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
mistralai/mistral-medium-3-5:none PASS 100% 201 in → 36 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 189 in → 42 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000"}}
mistralai/mistral-medium-3-5:xhigh PASS 100% 201 in → 966 out (982 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 189 in → 27,613 out (30,170 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k2.6:high PASS 100% 174 in → 2,361 out (2,614 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:high__json_schema PASS 100% 174 in → 1,176 out (1,135 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
moonshotai/kimi-k2.6:low PASS 100% 174 in → 1,863 out (1,839 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:low__json_schema PASS 100% 174 in → 1,524 out (1,624 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files" : { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
moonshotai/kimi-k2.6:medium PASS 100% 173 in → 2,120 out (2,094 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.6:medium__json_schema PASS 100% 173 in → 2,265 out (2,297 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
moonshotai/kimi-k2.6:xhigh PASS 100% 174 in → 3,315 out (3,289 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:high PASS 100% 174 in → 460 out (500 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:high__json_schema PASS 100% 266 in → 1,246 out (1,211 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k2.7-code:low PASS 100% 173 in → 291 out (265 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:medium PASS 100% 173 in → 508 out (480 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:xhigh PASS 100% 174 in → 333 out (307 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 266 in → 483 out (448 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k3:high PASS 100% 251 in → 396 out (355 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:high__json_schema PASS 100% 356 in → 401 out (353 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
moonshotai/kimi-k3:low PASS 100% 251 in → 552 out (511 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:low__json_schema PASS 100% 251 in → 1,045 out (1,001 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 9000 ```
moonshotai/kimi-k3:max PASS 100% 251 in → 597 out (558 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:max__json_schema PASS 100% 251 in → 594 out (555 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium PASS 100% 251 in → 304 out (265 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:medium__json_schema PASS 100% 251 in → 356 out (315 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh PASS 100% 251 in → 273 out (234 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 251 in → 735 out (694 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 202 in → 932 out (992 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 202 in → 1,078 out (1,116 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 202 in → 972 out (1,040 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 202 in → 257 out (247 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 202 in → 515 out (549 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 202 in → 29 out (0 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 202 in → 58 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 202 in → 954 out (989 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 202 in → 758 out (794 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 202 in → 260 out (218 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 202 in → 315 out (252 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 202 in → 405 out (356 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 202 in → 367 out (300 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 202 in → 256 out (211 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 202 in → 339 out (278 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 202 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 202 in → 51 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 202 in → 281 out (290 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 202 in → 387 out (316 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
openai/gpt-5.4-mini:high PASS 100% 169 in → 164 out (134 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:low PASS 100% 169 in → 122 out (92 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:medium PASS 100% 169 in → 124 out (94 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-mini:xhigh PASS 100% 169 in → 202 out (172 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-nano:high PASS 100% 169 in → 218 out (178 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:low PASS 100% 169 in → 38 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:medium PASS 100% 169 in → 38 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
openai/gpt-5.4-nano:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4-nano:xhigh PASS 100% 169 in → 514 out (484 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:high PASS 100% 169 in → 546 out (516 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:low PASS 100% 169 in → 47 out (17 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:medium PASS 100% 169 in → 242 out (212 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.4:xhigh PASS 100% 169 in → 546 out (516 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:high PASS 100% 169 in → 284 out (254 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:low PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:medium PASS 100% 169 in → 304 out (274 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.5:xhigh PASS 100% 169 in → 539 out (509 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:high PASS 100% 169 in → 187 out (157 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:low PASS 100% 169 in → 153 out (123 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:max PASS 100% 169 in → 114 out (84 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:medium PASS 100% 169 in → 137 out (107 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-luna:xhigh PASS 100% 169 in → 247 out (217 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:high PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:low PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:max PASS 100% 169 in → 171 out (141 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:medium PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-sol:xhigh PASS 100% 169 in → 126 out (96 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:high PASS 100% 169 in → 102 out (72 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:low PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:max PASS 100% 169 in → 273 out (243 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:medium PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:none PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-5.6-terra:xhigh PASS 100% 169 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high PASS 100% 230 in → 660 out (730 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:high__json_schema PASS 100% 232 in → 745 out (811 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-oss-120b:low PASS 100% 232 in → 386 out (405 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:low__json_schema PASS 100% 228 in → 158 out (124 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
openai/gpt-oss-120b:medium PASS 100% 217 in → 412 out (433 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:medium__json_schema PASS 100% 232 in → 298 out (280 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-oss-120b:xhigh PASS 100% 217 in → 644 out (716 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 232 in → 659 out (754 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
openai/gpt-oss-20b:high PASS 100% 230 in → 275 out (251 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:high__json_schema PASS 100% 217 in → 1,131 out (1,257 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
openai/gpt-oss-20b:low PASS 100% 228 in → 83 out (56 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:low__json_schema PASS 100% 230 in → 123 out (76 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"} }
openai/gpt-oss-20b:medium PASS 100% 232 in → 417 out (448 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
openai/gpt-oss-20b:xhigh PASS 100% 217 in → 572 out (624 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:high PASS 100% 191 in → 1,372 out (1,341 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:high__json_schema PASS 100% 191 in → 463 out (423 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-m.1:low PASS 100% 191 in → 632 out (601 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:low__json_schema PASS 100% 191 in → 2,051 out (2,013 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-m.1:medium PASS 100% 191 in → 269 out (238 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:medium__json_schema PASS 100% 191 in → 2,294 out (2,247 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-m.1:none PASS 100% 191 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:xhigh PASS 100% 191 in → 637 out (606 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-m.1:xhigh__json_schema PASS 100% 191 in → 403 out (360 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
poolside/laguna-xs-2.1:high PASS 100% 191 in → 1,553 out (1,522 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:high__json_schema PASS 100% 191 in → 510 out (461 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:low PASS 100% 191 in → 494 out (465 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:medium PASS 100% 191 in → 572 out (541 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 191 in → 551 out (511 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs-2.1:none PASS 100% 191 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:none__json_schema PASS 100% 191 in → 47 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs-2.1:xhigh PASS 100% 191 in → 438 out (409 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 191 in → 1,368 out (1,320 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
poolside/laguna-xs.2:high PASS 100% 228 in → 246 out (215 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:low__json_schema PASS 100% 228 in → 202 out (161 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:medium__json_schema PASS 100% 228 in → 253 out (212 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:none PASS 100% 228 in → 30 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
poolside/laguna-xs.2:none__json_schema PASS 100% 228 in → 38 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 228 in → 362 out (320 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-27b:high PASS 100% 185 in → 1,264 out (1,229 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:high__json_schema PASS 100% 185 in → 1,603 out (1,581 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-27b:low PASS 100% 185 in → 1,598 out (1,590 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:medium PASS 100% 185 in → 1,547 out (1,558 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-27b:medium__json_schema PASS 100% 185 in → 1,932 out (1,891 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-27b:none PASS 100% 187 in → 39 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ```
qwen/qwen3.6-27b:none__json_schema PASS 100% 187 in → 54 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-27b:xhigh PASS 100% 185 in → 1,329 out (1,331 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:high PASS 100% 185 in → 1,441 out (1,483 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 185 in → 1,626 out (1,554 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:low PASS 100% 185 in → 1,790 out (1,757 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 185 in → 2,492 out (2,512 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
qwen/qwen3.6-35b-a3b:medium PASS 100% 185 in → 1,918 out (1,904 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 185 in → 1,651 out (1,607 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
qwen/qwen3.6-35b-a3b:none PASS 100% 187 in → 29 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 185 in → 1,822 out (1,780 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
qwen/qwen3.6-flash:high PASS 100% 185 in → 1,795 out (1,762 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:low PASS 100% 185 in → 2,505 out (2,472 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:medium PASS 100% 185 in → 1,722 out (1,689 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:none PASS 100% 187 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.6-flash:xhigh PASS 100% 185 in → 1,873 out (1,840 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:high PASS 100% 185 in → 1,662 out (1,629 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:low PASS 100% 185 in → 1,834 out (1,799 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:max PASS 100% 185 in → 1,322 out (1,289 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:medium PASS 100% 185 in → 1,433 out (1,398 reasoning)
--- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:none PASS 100% 187 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-flash:xhigh PASS 100% 185 in → 1,903 out (1,870 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:high PASS 100% 185 in → 295 out (262 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:low PASS 100% 185 in → 790 out (757 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:medium PASS 100% 185 in → 233 out (200 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:none PASS 100% 187 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-max:xhigh PASS 100% 185 in → 889 out (856 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:high PASS 100% 185 in → 432 out (399 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:low PASS 100% 185 in → 1,213 out (1,180 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:medium PASS 100% 185 in → 446 out (413 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:none PASS 100% 187 in → 28 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
qwen/qwen3.7-plus:xhigh PASS 100% 185 in → 1,522 out (1,489 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:high PASS 100% 181 in → 1,744 out (1,717 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:high__json_schema PASS 100% 181 in → 1,395 out (1,470 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:low PASS 100% 181 in → 762 out (749 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:low__json_schema PASS 100% 181 in → 1,298 out (1,266 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:medium PASS 100% 181 in → 1,112 out (1,086 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:medium__json_schema PASS 100% 181 in → 1,119 out (1,117 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:none PASS 100% 184 in → 26 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:none__json_schema PASS 100% 184 in → 44 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
tencent/hy3:xhigh PASS 100% 181 in → 1,981 out (1,955 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
tencent/hy3:xhigh__json_schema PASS 100% 181 in → 1,790 out (1,835 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling-small:high PASS 100% 178 in → 690 out (660 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:high__json_schema PASS 100% 178 in → 571 out (526 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
thinkingmachines/inkling-small:low PASS 100% 178 in → 110 out (80 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:low__json_schema PASS 100% 178 in → 154 out (105 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" }}
thinkingmachines/inkling-small:max PASS 100% 178 in → 822 out (792 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:max__json_schema PASS 100% 178 in → 664 out (619 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
thinkingmachines/inkling-small:medium PASS 100% 178 in → 256 out (256 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:none PASS 100% 176 in → 27 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:xhigh PASS 100% 178 in → 830 out (862 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling-small:xhigh__json_schema PASS 100% 178 in → 481 out (433 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
thinkingmachines/inkling:high PASS 100% 178 in → 530 out (500 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:low PASS 100% 178 in → 181 out (178 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:max PASS 100% 178 in → 716 out (770 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:max__json_schema PASS 100% 178 in → 761 out (720 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
thinkingmachines/inkling:medium PASS 100% 178 in → 306 out (275 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
thinkingmachines/inkling:xhigh PASS 100% 178 in → 422 out (392 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:high PASS 100% 346 in → 929 out (905 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:high__json_schema PASS 100% 412 in → 1,346 out (1,304 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:low PASS 100% 352 in → 431 out (407 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:low__json_schema PASS 100% 418 in → 554 out (510 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.3:max PASS 100% 346 in → 882 out (858 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:max__json_schema PASS 100% 412 in → 929 out (899 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")","utils.py":"DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:medium PASS 100% 352 in → 752 out (728 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:medium__json_schema PASS 100% 418 in → 509 out (467 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.3:none PASS 100% 344 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:none__json_schema PASS 100% 410 in → 34 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000"}}
x-ai/grok-4.3:xhigh PASS 100% 346 in → 468 out (444 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.3:xhigh__json_schema PASS 100% 412 in → 1,155 out (1,113 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
x-ai/grok-4.5:high PASS 100% 375 in → 380 out (354 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:high__json_schema PASS 100% 451 in → 1,002 out (953 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:low PASS 100% 375 in → 268 out (242 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:low__json_schema PASS 100% 451 in → 586 out (537 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:max PASS 100% 375 in → 542 out (516 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:max__json_schema PASS 100% 451 in → 781 out (732 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:medium PASS 100% 375 in → 535 out (509 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:medium__json_schema PASS 100% 451 in → 646 out (597 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
x-ai/grok-4.5:xhigh PASS 100% 375 in → 354 out (328 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
x-ai/grok-4.5:xhigh__json_schema PASS 100% 451 in → 731 out (682 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-4.7-flash:high PASS 100% 166 in → 515 out (503 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:high__json_schema PASS 100% 166 in → 627 out (580 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-4.7-flash:low PASS 100% 166 in → 2,729 out (2,861 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:medium PASS 100% 166 in → 538 out (506 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:none PASS 100% 166 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:none__json_schema PASS 100% 166 in → 45 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-4.7-flash:xhigh PASS 100% 166 in → 525 out (550 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-4.7-flash:xhigh__json_schema PASS 100% 166 in → 641 out (674 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-5.2:high PASS 100% 173 in → 655 out (702 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:high__json_schema PASS 100% 173 in → 544 out (495 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-5.2:low PASS 100% 173 in → 598 out (592 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:low__json_schema PASS 100% 173 in → 962 out (1,027 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
z-ai/glm-5.2:medium PASS 100% 173 in → 978 out (953 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:medium__json_schema PASS 100% 358 in → 664 out (616 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-5.2:none PASS 100% 167 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:none__json_schema PASS 100% 167 in → 43 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
z-ai/glm-5.2:xhigh PASS 100% 173 in → 1,207 out (1,327 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
z-ai/glm-5.2:xhigh__json_schema PASS 100% 358 in → 539 out (503 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) {"files":{"main.py":"def main():\n print(\"Running enterprise\")\n","utils.py":"DEBUG = True\nPORT = 9000\n"}}
anthropic/claude-haiku-4.5:none FAIL 50% 203 in → 31 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:high FAIL 50% 203 in → 408 out (173 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Reasoning:** - **main.py**: "Running enterprise" (branch B) is more descriptive than the generic "Running v2" (branch A). - **utils.py**: `DEBUG = True` (branch B) is the safer choice (keeps debugging enabled), and `PORT = 9000` is taken along with it as the consistent branch B pairing.
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:low FAIL 50% 203 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:max FAIL 50% 203 in → 638 out (212 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Reasoning:** - **main.py:** "Running enterprise" (branch B) is more descriptive/specific than the generic "Running v2" (branch A). - **utils.py:** `DEBUG = True` (branch B) is the safer choice (ensures debug info is available), paired with its corresponding `PORT = 9000`.
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:medium FAIL 50% 203 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:none FAIL 50% 203 in → 39 out (0 reasoning)
main.py ``` def main(): print("Running enterprise") ``` utils.py ``` DEBUG = True PORT = 8000 ```
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.6:xhigh FAIL 50% 203 in → 578 out (207 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Reasoning:** - **main.py:** "Running enterprise" (branch B) is more descriptive/specific than the generic "Running v2" (branch A). - **utils.py:** Chose branch B's block — `DEBUG = True` is the safer option (enables debug logging/error output), and `PORT = 9000` comes with it as part of that branch's configuration.
Failure: Content mismatch: ['utils.py']
anthropic/claude-opus-4.8:none FAIL 50% 306 in → 144 out (0 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution notes:** - **main.py**: Kept `"Running enterprise"` (branch B) as the more descriptive version. - **utils.py**: Kept `DEBUG = True` and `PORT = 9000` (branch B) for the safer DEBUG setting.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:high FAIL 50% 203 in → 619 out (176 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution rationale:** - **main.py**: `"Running enterprise"` (HEAD) is chosen over `"Running v2"` (feature) as it is the more descriptive and meaningful label. - **utils.py**: `DEBUG = True` (HEAD) is chosen as the safer default since it preserves debug visibility. `PORT = 9000` (HEAD) is kept alongside it, as both values come from the same HEAD state and the user's preference was anchored to the HEAD branch's safer `DEBUG` setting.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:max FAIL 50% 203 in → 724 out (266 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution notes:** - **main.py** — Chose `"Running enterprise"` (HEAD/branch B) over `"Running v2"` as it is more descriptive and meaningful to a reader. - **utils.py** — Chose `DEBUG = True` (HEAD/branch B) as the safer default for visibility into runtime behaviour. `PORT = 9000` is kept from the same branch to remain internally consistent, since mixing PORT from one branch and DEBUG from another could introduce unintended configuration mismatches.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:medium FAIL 50% 203 in → 527 out (202 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution reasoning:** - **main.py** — `"Running enterprise"` (HEAD/branch B) is chosen over `"Running v2"` as it is more descriptive, clearly communicating the deployment tier rather than a generic version tag. - **utils.py** — `DEBUG = True` is chosen per the instruction to prefer the safer value (leaving debug output enabled avoids silently swallowing errors). `PORT = 9000` follows from the same branch (HEAD/branch B) that carried `DEBUG = True`, keeping the configuration internally consistent.
Failure: Content mismatch: ['utils.py']
anthropic/claude-sonnet-4.6:xhigh FAIL 50% 203 in → 700 out (281 reasoning)
main.py ```python def main(): print("Running enterprise") ``` utils.py ```python DEBUG = True PORT = 9000 ``` **Resolution notes:** - **main.py** → Chose `"Running enterprise"` (HEAD) over `"Running v2"` (feature) as it is more descriptive, indicating the deployment tier rather than a generic version tag. - **utils.py** → Chose `DEBUG = True` (HEAD) as it is the safer default, preventing silent failures in an unknown environment. `PORT = 9000` follows from the same HEAD branch for consistency, keeping the configuration set coherent.
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash-0731:none__json_schema FAIL 50% 178 in → 45 out (0 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n" } }
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-flash:none FAIL 50% 178 in → 24 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
deepseek/deepseek-v4-pro:xhigh FAIL 50% 178 in → 25 out (0 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['utils.py']
inclusionai/ling-2.6-flash FAIL 50% 205 in → 32 out
main.py def main(): print("Running enterprise v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
main.py: def main(): print("Running enterprise") print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
liquid/lfm-2-24b-a2b FAIL 50% 201 in → 27 out
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
poolside/laguna-xs-2.1:low__json_schema FAIL 50% 191 in → 1,390 out (1,344 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 9000" } }
Failure: Content mismatch: ['main.py']
poolside/laguna-xs.2:low FAIL 50% 228 in → 219 out (186 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
poolside/laguna-xs.2:xhigh FAIL 50% 228 in → 291 out (254 reasoning)
main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
qwen/qwen3.6-35b-a3b:xhigh FAIL 50% 185 in → 1,938 out (1,950 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
thinkingmachines/inkling:none FAIL 50% 176 in → 29 out (0 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
Failure: Content mismatch: ['main.py']
thinkingmachines/inkling:none__json_schema FAIL 50% 176 in → 46 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running v2\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
Failure: Content mismatch: ['main.py']
anthropic/claude-fable-5:high__json_schema FAIL 0% 539 in → 129 out (59 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:low__json_schema FAIL 0% 539 in → 48 out (31 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:max__json_schema FAIL 0% 539 in → 811 out (105 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:medium__json_schema FAIL 0% 539 in → 50 out (45 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 539 in → 407 out (141 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:high__json_schema FAIL 0% 409 in → 1,191 out (964 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:low__json_schema FAIL 0% 409 in → 740 out (598 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:medium__json_schema FAIL 0% 409 in → 719 out (514 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:none__json_schema FAIL 0% 379 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-haiku-4.5:xhigh__json_schema FAIL 0% 409 in → 3,090 out (1,096 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:high__json_schema FAIL 0% 380 in → 304 out (183 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:low__json_schema FAIL 0% 380 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:max__json_schema FAIL 0% 380 in → 451 out (198 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:medium__json_schema FAIL 0% 380 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:none__json_schema FAIL 0% 380 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.6:xhigh__json_schema FAIL 0% 380 in → 359 out (253 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:high__json_schema FAIL 0% 544 in → 16 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:low__json_schema FAIL 0% 544 in → 16 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:max__json_schema FAIL 0% 544 in → 691 out (129 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:medium__json_schema FAIL 0% 544 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:none__json_schema FAIL 0% 544 in → 11 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.7:xhigh__json_schema FAIL 0% 544 in → 110 out (48 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 539 in → 123 out (71 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 539 in → 10 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 539 in → 906 out (128 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": {}}
Failure: No file blocks found
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 539 in → 53 out (38 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 539 in → 8 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 539 in → 448 out (114 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:high__json_schema FAIL 0% 380 in → 349 out (188 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:low__json_schema FAIL 0% 380 in → 84 out (62 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:max__json_schema FAIL 0% 380 in → 645 out (200 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:medium__json_schema FAIL 0% 380 in → 313 out (228 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:none__json_schema FAIL 0% 380 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-4.6:xhigh__json_schema FAIL 0% 380 in → 640 out (264 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 539 in → 100 out (65 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:low__json_schema FAIL 0% 539 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 539 in → 15 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:none__json_schema FAIL 0% 539 in → 14 out (0 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 539 in → 336 out (51 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
arcee-ai/trinity-mini:low FAIL 0% 173 in → 994 out (1,107 reasoning)
def main(): print("Running enterprise") DEBUG = True PORT = 9000
Failure: No file blocks found
arcee-ai/trinity-mini:medium FAIL 0% 173 in → 283 out (318 reasoning)
Runningenterprise DEBUG = True PORT = 9000
Failure: No file blocks found
arcee-ai/trinity-mini:xhigh FAIL 0% 173 in → 224 out (240 reasoning)
main.pydef main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
deepseek/deepseek-v4-flash-0731:xhigh__json_schema FAIL 0% 178 in → 1,158 out (1,206 reasoning)
main.py: --- main.py def main(): print("Running v2") utils.py: --- utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": {"main.py": "--- main.py\ndef main():\n print(\"Running v2\")", "utils.py": "--- utils.py\nDEBUG = True\nPORT = 9000"} }
Failure: Duplicate file block: main.py
Invalid JSON. Output: --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none__json_schema FAIL 0% 258 in → 43 out (0 reasoning)
main.py: def main(): print("Running enterprise v2") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running enterprise v2\")\n", "utils.py": "DEBUG = True\nPORT = 8000\n"}}
Failure: Content mismatch: ['main.py', 'utils.py']
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:medium__json_schema FAIL 0% 180 in → 398 out (389 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "files": {}}
Failure: No file blocks found
google/gemini-3.1-flash-lite-preview:low__json_schema FAIL 0% 272 in → 186 out (124 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")", "utils.py": "utils.py\nDEBUG = True\nPORT = 9000" } }
Failure: Duplicate file block: main.py
google/gemini-3.1-pro-preview:low__json_schema FAIL 0% 272 in → 832 out (795 reasoning)
main.py: def main(): print("Running enterprise") utils.py: DEBUG = True; PORT = 9000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main(): print(\"Running enterprise\")", "utils.py": "DEBUG = True; PORT = 9000"}}
Failure: Content mismatch: ['main.py', 'utils.py']
google/gemma-4-31b-it:high__json_schema FAIL 0% 206 in → 309 out (275 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files": { } }
Failure: No file blocks found
google/gemma-4-31b-it:xhigh__json_schema FAIL 0% 209 in → 630 out (624 reasoning)
(empty output)
JSON Schema Structured Output
(raw) {"files":{}}
Failure: No file blocks found
: main.py: def main(): print("Running enterprise") utils.py: DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "def main():\n print(\"Running enterprise\")\n", "" : "", "utils.py": "DEBUG = True\nPORT = 9000\n" } }
Failure: Expected file heading before file content
Invalid JSON. Output: main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
mistralai/devstral-2512 FAIL 0% 189 in → 36 out
Here are the resolved file contents: --- main.py def main(): print("Running enterprise") --- utils.py DEBUG = True PORT = 9000
Failure: Expected file heading before file content
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 2 (char 1)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}', 'provider_name': 'Azure', 'is_byok': False, 'previous_errors': [{'code': 400, 'message': 'Provider returned error', 'provider_name': 'OpenAI', 'raw': '{\n "error": {\n "message": "Invalid schema for response_format \'ResolvedFileBlocks\': In context=(), \'required\' is required to be supplied and to be an array including every key in properties. Extra required key \'files\' supplied.",\n "type": "invalid_request_error",\n "param": "text.format.schema",\n "code": "invalid_json_schema"\n }\n}'}]}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
openai/gpt-oss-20b:medium__json_schema FAIL 0% 230 in → 154 out (0 reasoning)
. We have: branch A says 'Running v2', branch B says 'Running enterprise'. The instruction: : Running enterprise analysis: We need to provide resolved content of main.py and utils.py. For main.py: we prefer more descriptive version. Branch A says 'Running v2', branch B says 'Running enterprise'. Which is more descriptive? Probably 'Running enterprise' is more descriptive? The instruction: prefer the more descriptive version: Running enterprise preferred: v2:
JSON Schema Structured Output
(raw) {"files": {"analysis":"We need to provide resolved content of main.py and utils.py. For main.py: we prefer more descriptive version. Branch A says 'Running v2', branch B says 'Running enterprise'. Which is more descriptive? Probably 'Running enterprise' is more descriptive? The instruction: " ,". We have: branch A says 'Running v2', branch B says 'Running enterprise'. The instruction: " : "Running enterprise" , "v2" : "" , "prefer the more descriptive version" : " Running enterprise" , "preferred" : " " }}
Failure: No file blocks found
openai/gpt-oss-20b:xhigh__json_schema FAIL 0% 230 in → 124 out (0 reasoning)
analysis: We need to provide resolved file contents. For main.py: the branch giving more descriptive version is 'Running enterprise' (descriptive). So choose that. For utils.py: safer DEBUG=True preference; keep DEBUG=True and PORT=9000. Provide two resolved file contents separated by blank line with each file prefixed by its filename. So format: filename line then file content. Let's produce: main.py ... utils.py ... Use the content: main.py with def main(): print(
JSON Schema Structured Output
(raw) {"files": {"analysis": "We need to provide resolved file contents. For main.py: the branch giving more descriptive version is 'Running enterprise' (descriptive). So choose that. For utils.py: safer DEBUG=True preference; keep DEBUG=True and PORT=9000. Provide two resolved file contents separated by blank line with each file prefixed by its filename. So format: filename line then file content. Let's produce: \nmain.py\n...\n\nutils.py\n...\n Use the content: main.py with def main(): print(" } }
Failure: Expected file heading before file content
poolside/laguna-m.1:none__json_schema FAIL 0% 191 in → 55 out (0 reasoning)
main.py: Resolved content preferring 'Running enterprise' (more descriptive) utils.py: Resolved content keeping DEBUG=True and PORT=9000 (safer defaults)
JSON Schema Structured Output
(raw) { "files": { "main.py": "Resolved content preferring 'Running enterprise' (more descriptive)", "utils.py": "Resolved content keeping DEBUG=True and PORT=9000 (safer defaults)" } }
Failure: Content mismatch: ['main.py', 'utils.py']
poolside/laguna-xs.2:high__json_schema FAIL 0% 228 in → 203 out (192 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "files": {} }
Failure: No file blocks found
poolside/laguna-xs.2:medium FAIL 0% 228 in → 439 out (407 reasoning)
main.py def main(): print("Running v2") utils.py DEBUG = True PORT = 8000
Failure: Content mismatch: ['main.py', 'utils.py']
qwen/qwen3.6-27b:low__json_schema FAIL 0% 185 in → 1,225 out (1,216 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "files": { }}
Failure: No file blocks found
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 187 in → 40 out (0 reasoning)
main.py: def main(): print("Running v2") utils.py: DEBUG = True PORT = 8000
JSON Schema Structured Output
(raw) {"files": {"main.py": "def main():\n print(\"Running v2\")", "utils.py": "DEBUG = True\nPORT = 8000"}}
Failure: Content mismatch: ['main.py', 'utils.py']
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-38521ca3-d1f5-909f-9d57-6b5e5e362eb8","request_id":"38521ca3-d1f5-909f-9d57-6b5e5e362eb8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9733ad85-1189-93be-8e15-bb93cbe8c29a","request_id":"9733ad85-1189-93be-8e15-bb93cbe8c29a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-14d92be9-95a4-986a-be79-636bf71298cc","request_id":"14d92be9-95a4-986a-be79-636bf71298cc"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-98cd038e-999c-9699-a2aa-3c66f6da5b8f","request_id":"98cd038e-999c-9699-a2aa-3c66f6da5b8f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d2461513-c200-9604-9e60-207625aed8ff","request_id":"d2461513-c200-9604-9e60-207625aed8ff"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-f72f07f7-7b72-93ca-876c-7ade1a30edbd"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-8e542617-b059-9e2b-97a0-f99152d99d54"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-6439ab76-d88f-9979-89bc-146587cbc1f2"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-29aee644-82e9-9e3f-8455-4f3863fbbf9d"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-4f35bcc2-ed3a-9773-abe9-41139cd13ffc"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-70fe25df-41af-9466-8b15-13f733feb187"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8811e51a-830b-9b63-8af6-94d545b1c0d5","request_id":"8811e51a-830b-9b63-8af6-94d545b1c0d5"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-3c792ceb-a57b-9af0-9cdf-343af65f840a","request_id":"3c792ceb-a57b-9af0-9cdf-343af65f840a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-82a57fcd-2508-9a79-8d17-b049fcb2577e","request_id":"82a57fcd-2508-9a79-8d17-b049fcb2577e"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-16ea9c39-3e84-93f2-a1e3-3c35738f1e01","request_id":"16ea9c39-3e84-93f2-a1e3-3c35738f1e01"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8d50db60-7426-92b0-a73d-1a58306809a3","request_id":"8d50db60-7426-92b0-a73d-1a58306809a3"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-6594c7a5-8aab-9a4b-b3d1-56911085f4b4","request_id":"6594c7a5-8aab-9a4b-b3d1-56911085f4b4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-e99b72fb-5c7e-97c5-99cf-98f8326cc5d4","request_id":"e99b72fb-5c7e-97c5-99cf-98f8326cc5d4"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-8cc6b406-95b0-9850-a157-58fb61318894","request_id":"8cc6b406-95b0-9850-a157-58fb61318894"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-d0b5eceb-415c-970d-9eb8-5de05bd918ab","request_id":"d0b5eceb-415c-970d-9eb8-5de05bd918ab"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-34c06054-9638-93ed-9e89-81ad5d99ed25","request_id":"34c06054-9638-93ed-9e89-81ad5d99ed25"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:medium__json_schema FAIL 0% 178 in → 474 out (423 reasoning)
main.py: main.py def main(): print("Running enterprise") utils.py: utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
(raw) { "files": { "main.py": "main.py\ndef main():\n print(\"Running enterprise\")", "utils.py": "utils.py\nDEBUG = True\nPORT = 9000" } }
Failure: Duplicate file block: main.py
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000{"files": {"main.py": "def main():\n print(\"Running enterprise\")\n", "utils.py": "DEBUG = True\nPORT = 9000\n"}}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output: main.py def main(): print("Running enterprise") utils.py DEBUG = True PORT = 9000
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)