Search with line numbers using git grep -n
Tests ability to search with line numbers using git grep -n. Evaluates output-format options for search results.

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 mkdir -p src
  5. 05 printf 'import os import sys import logging logger = logging.getLogger(__name__) def main(): logger.info("Starting") os.environ["MODE"] = "prod" sys.exit(0) if __name__ == "__main__": main() ' > src/app.py
  6. 06 git add .
  7. 07 git commit -m 'Add application entry point'
  8. 08 echo 'git grep -n import' > .grep_command
  9. 09 git add .grep_command
  10. 10 git commit -m 'Add grep sentinel'
Prompt
Here is the output of a git grep -n command showing line numbers. On which line numbers does 'import' appear in src/app.py? Output ONLY the line numbers separated by commas, nothing else.
Expected
1,2,3

Scoped model quality, cost, API time, and token usage for git_grep/f004.

Loading...
Loading raw attempt evidence…
anthropic/claude-fable-5:max PASS 100% 103 in → 129 out (34 reasoning)
1,2,3
anthropic/claude-haiku-4.5:high PASS 100% 115 in → 118 out (99 reasoning)
1,2,3
anthropic/claude-haiku-4.5:high__json_schema PASS 100% 288 in → 205 out (160 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-haiku-4.5:low PASS 100% 115 in → 128 out (101 reasoning)
1,2,3
anthropic/claude-haiku-4.5:low__json_schema PASS 100% 288 in → 216 out (165 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-haiku-4.5:medium PASS 100% 115 in → 138 out (105 reasoning)
1,2,3
anthropic/claude-haiku-4.5:medium__json_schema PASS 100% 288 in → 345 out (303 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-haiku-4.5:none PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-haiku-4.5:none__json_schema PASS 100% 258 in → 15 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-haiku-4.5:xhigh PASS 100% 115 in → 125 out (98 reasoning)
1,2,3
anthropic/claude-haiku-4.5:xhigh__json_schema PASS 100% 288 in → 281 out (245 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-opus-4.6:high PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:high__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.6:low PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:low__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.6:max PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:max__json_schema PASS 100% 259 in → 90 out (56 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.6:medium PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:medium__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.6:none PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:none__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.6:xhigh PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.6:xhigh__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:high PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:high__json_schema PASS 100% 339 in → 20 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:low PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:low__json_schema PASS 100% 339 in → 20 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:max PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:max__json_schema PASS 100% 339 in → 21 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:medium PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:medium__json_schema PASS 100% 339 in → 20 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:none PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:none__json_schema PASS 100% 339 in → 20 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-opus-4.7:xhigh PASS 100% 108 in → 10 out (0 reasoning)
1,2,3
anthropic/claude-opus-4.7:xhigh__json_schema PASS 100% 339 in → 20 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:high PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:high__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:low PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:low__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:max PASS 100% 86 in → 104 out (66 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:max__json_schema PASS 100% 259 in → 92 out (55 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:medium PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:medium__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:none PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:none__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-4.6:xhigh PASS 100% 86 in → 9 out (0 reasoning)
1,2,3
anthropic/claude-sonnet-4.6:xhigh__json_schema PASS 100% 259 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-5:low__json_schema PASS 100% 334 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
anthropic/claude-sonnet-5:none__json_schema PASS 100% 334 in → 17 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
anthropic/claude-sonnet-5:xhigh PASS 100% 103 in → 71 out (45 reasoning)
1,2,3
arcee-ai/trinity-large-thinking:high PASS 100% 78 in → 239 out (233 reasoning)
1,2,3
arcee-ai/trinity-large-thinking:low PASS 100% 78 in → 216 out (208 reasoning)
1,2,3
arcee-ai/trinity-large-thinking:medium PASS 100% 78 in → 221 out (213 reasoning)
1,2,3
arcee-ai/trinity-large-thinking:xhigh PASS 100% 78 in → 187 out (179 reasoning)
1,2,3
arcee-ai/trinity-mini:high PASS 100% 78 in → 293 out (305 reasoning)
1,2,3
arcee-ai/trinity-mini:high__json_schema PASS 100% 78 in → 340 out (346 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers":"1,2,3" }
arcee-ai/trinity-mini:low PASS 100% 78 in → 280 out (277 reasoning)
1,2,3
arcee-ai/trinity-mini:low__json_schema PASS 100% 78 in → 341 out (330 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
arcee-ai/trinity-mini:medium PASS 100% 78 in → 325 out (341 reasoning)
1,2,3
arcee-ai/trinity-mini:medium__json_schema PASS 100% 78 in → 319 out (318 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
arcee-ai/trinity-mini:xhigh PASS 100% 78 in → 270 out (268 reasoning)
1,2,3
arcee-ai/trinity-mini:xhigh__json_schema PASS 100% 78 in → 302 out (296 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:high PASS 100% 151 in → 28 out (22 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:high__json_schema PASS 100% 157 in → 46 out (27 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:low PASS 100% 72 in → 140 out (123 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:low__json_schema PASS 100% 72 in → 86 out (64 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:max PASS 100% 164 in → 143 out (137 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:max__json_schema PASS 100% 151 in → 65 out (49 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:medium PASS 100% 72 in → 51 out (37 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:medium__json_schema PASS 100% 72 in → 64 out (46 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:none PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:none__json_schema PASS 100% 72 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash-0731:xhigh PASS 100% 72 in → 27 out (22 reasoning)
1,2,3
deepseek/deepseek-v4-flash-0731:xhigh__json_schema PASS 100% 157 in → 75 out (60 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash:high PASS 100% 72 in → 57 out (50 reasoning)
1,2,3
deepseek/deepseek-v4-flash:high__json_schema PASS 100% 72 in → 52 out (30 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash:low PASS 100% 72 in → 60 out (53 reasoning)
1,2,3
deepseek/deepseek-v4-flash:low__json_schema PASS 100% 74 in → 168 out (152 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash:medium PASS 100% 72 in → 122 out (115 reasoning)
1,2,3
deepseek/deepseek-v4-flash:medium__json_schema PASS 100% 72 in → 84 out (68 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash:none PASS 100% 72 in → 5 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 72 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-flash:xhigh PASS 100% 72 in → 47 out (33 reasoning)
1,2,3
deepseek/deepseek-v4-flash:xhigh__json_schema PASS 100% 151 in → 135 out (123 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-pro:high PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-pro:high__json_schema PASS 100% 72 in → 14 out (13 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-pro:low PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-pro:low__json_schema PASS 100% 72 in → 120 out (99 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-pro:medium PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-pro:medium__json_schema PASS 100% 74 in → 102 out (86 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
deepseek/deepseek-v4-pro:none PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-pro:xhigh PASS 100% 72 in → 6 out (0 reasoning)
1,2,3
deepseek/deepseek-v4-pro:xhigh__json_schema PASS 100% 346 in → 1,349 out (1,336 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3-flash-preview:high PASS 100% 76 in → 166 out (161 reasoning)
1,2,3
google/gemini-3-flash-preview:high__json_schema PASS 100% 77 in → 254 out (243 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3-flash-preview:low PASS 100% 76 in → 158 out (153 reasoning)
1,2,3
google/gemini-3-flash-preview:low__json_schema PASS 100% 143 in → 230 out (219 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3-flash-preview:medium PASS 100% 76 in → 281 out (276 reasoning)
1,2,3
google/gemini-3-flash-preview:medium__json_schema PASS 100% 143 in → 286 out (275 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3-flash-preview:xhigh PASS 100% 76 in → 145 out (140 reasoning)
1,2,3
google/gemini-3-flash-preview:xhigh__json_schema PASS 100% 77 in → 243 out (232 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.1-flash-lite-preview:high PASS 100% 77 in → 177 out (172 reasoning)
1,2,3
google/gemini-3.1-flash-lite-preview:high__json_schema PASS 100% 77 in → 339 out (328 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.1-flash-lite-preview:low PASS 100% 77 in → 134 out (129 reasoning)
1,2,3
google/gemini-3.1-flash-lite-preview:low__json_schema PASS 100% 77 in → 138 out (126 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.1-flash-lite-preview:medium PASS 100% 76 in → 176 out (171 reasoning)
1,2,3
google/gemini-3.1-flash-lite-preview:medium__json_schema PASS 100% 77 in → 284 out (272 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.1-flash-lite-preview:xhigh PASS 100% 76 in → 254 out (249 reasoning)
1,2,3
google/gemini-3.1-flash-lite-preview:xhigh__json_schema PASS 100% 77 in → 328 out (311 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemini-3.1-pro-preview:high PASS 100% 76 in → 371 out (366 reasoning)
1,2,3
google/gemini-3.1-pro-preview:low PASS 100% 76 in → 349 out (344 reasoning)
1,2,3
google/gemini-3.1-pro-preview:low__json_schema PASS 100% 143 in → 12 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.1-pro-preview:medium PASS 100% 76 in → 363 out (358 reasoning)
1,2,3
google/gemini-3.1-pro-preview:xhigh PASS 100% 76 in → 289 out (284 reasoning)
1,2,3
google/gemini-3.1-pro-preview:xhigh__json_schema PASS 100% 143 in → 496 out (485 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.5-flash-lite:high PASS 100% 76 in → 307 out (302 reasoning)
1,2,3
google/gemini-3.5-flash-lite:high__json_schema PASS 100% 143 in → 484 out (473 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.5-flash-lite:low PASS 100% 77 in → 5 out (0 reasoning)
1,2,3
google/gemini-3.5-flash-lite:low__json_schema PASS 100% 143 in → 17 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemini-3.5-flash-lite:max PASS 100% 77 in → 318 out (313 reasoning)
1,2,3
google/gemini-3.5-flash-lite:max__json_schema PASS 100% 77 in → 464 out (453 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.5-flash-lite:medium PASS 100% 77 in → 255 out (250 reasoning)
1,2,3
google/gemini-3.5-flash-lite:medium__json_schema PASS 100% 77 in → 211 out (199 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.5-flash-lite:xhigh PASS 100% 77 in → 348 out (343 reasoning)
1,2,3
google/gemini-3.5-flash-lite:xhigh__json_schema PASS 100% 77 in → 330 out (318 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.5-flash:high PASS 100% 76 in → 295 out (290 reasoning)
1,2,3
google/gemini-3.5-flash:high__json_schema PASS 100% 143 in → 361 out (349 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.5-flash:low PASS 100% 76 in → 250 out (245 reasoning)
1,2,3
google/gemini-3.5-flash:low__json_schema PASS 100% 143 in → 142 out (130 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.5-flash:medium PASS 100% 76 in → 361 out (356 reasoning)
1,2,3
google/gemini-3.5-flash:medium__json_schema PASS 100% 143 in → 218 out (206 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
google/gemini-3.5-flash:xhigh PASS 100% 76 in → 429 out (424 reasoning)
1,2,3
google/gemini-3.5-flash:xhigh__json_schema PASS 100% 143 in → 229 out (212 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemini-3.6-flash:high PASS 100% 76 in → 465 out (460 reasoning)
1,2,3
google/gemini-3.6-flash:high__json_schema PASS 100% 77 in → 474 out (463 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.6-flash:low PASS 100% 77 in → 229 out (224 reasoning)
1,2,3
google/gemini-3.6-flash:low__json_schema PASS 100% 143 in → 11 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.6-flash:max PASS 100% 77 in → 380 out (375 reasoning)
1,2,3
google/gemini-3.6-flash:medium PASS 100% 77 in → 371 out (366 reasoning)
1,2,3
google/gemini-3.6-flash:medium__json_schema PASS 100% 77 in → 354 out (343 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemini-3.6-flash:xhigh PASS 100% 77 in → 268 out (263 reasoning)
1,2,3
google/gemini-3.6-flash:xhigh__json_schema PASS 100% 77 in → 367 out (356 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
google/gemma-4-26b-a4b-it:high PASS 100% 91 in → 304 out (225 reasoning)
1,2,3
google/gemma-4-26b-a4b-it:high__json_schema PASS 100% 91 in → 308 out (225 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-26b-a4b-it:low__json_schema PASS 100% 91 in → 226 out (155 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-26b-a4b-it:medium PASS 100% 92 in → 320 out (244 reasoning)
1,2,3
google/gemma-4-26b-a4b-it:medium__json_schema PASS 100% 92 in → 277 out (203 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-26b-a4b-it:none PASS 100% 89 in → 6 out (0 reasoning)
1,2,3
google/gemma-4-26b-a4b-it:none__json_schema PASS 100% 89 in → 15 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-26b-a4b-it:xhigh PASS 100% 92 in → 186 out (138 reasoning)
1,2,3
google/gemma-4-26b-a4b-it:xhigh__json_schema PASS 100% 91 in → 248 out (177 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-31b-it:high PASS 100% 95 in → 170 out (164 reasoning)
1,2,3
google/gemma-4-31b-it:low PASS 100% 92 in → 158 out (105 reasoning)
1,2,3
google/gemma-4-31b-it:low__json_schema PASS 100% 91 in → 186 out (125 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-31b-it:medium PASS 100% 95 in → 155 out (149 reasoning)
1,2,3
google/gemma-4-31b-it:medium__json_schema PASS 100% 91 in → 180 out (125 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-31b-it:none PASS 100% 89 in → 6 out (0 reasoning)
1,2,3
google/gemma-4-31b-it:none__json_schema PASS 100% 89 in → 15 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
google/gemma-4-31b-it:xhigh PASS 100% 92 in → 156 out (108 reasoning)
1,2,3
google/gemma-4-31b-it:xhigh__json_schema PASS 100% 91 in → 161 out (106 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
ibm-granite/granite-4.1-8b PASS 100% 86 in → 6 out
1,2,3
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
inclusionai/ling-2.6-flash PASS 100% 88 in → 6 out
1,2,3
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
minimax/minimax-m2.5:high PASS 100% 106 in → 165 out (158 reasoning)
1,2,3
minimax/minimax-m2.5:high__json_schema PASS 100% 117 in → 640 out (627 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
minimax/minimax-m2.5:low PASS 100% 106 in → 193 out (188 reasoning)
1,2,3
minimax/minimax-m2.5:low__json_schema PASS 100% 106 in → 142 out (126 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
minimax/minimax-m2.5:medium PASS 100% 106 in → 92 out (84 reasoning)
1,2,3
minimax/minimax-m2.5:medium__json_schema PASS 100% 106 in → 196 out (182 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
minimax/minimax-m2.5:xhigh PASS 100% 106 in → 154 out (153 reasoning)
1,2,3
minimax/minimax-m2.5:xhigh__json_schema PASS 100% 117 in → 382 out (384 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
minimax/minimax-m2.7:high PASS 100% 106 in → 131 out (129 reasoning)
1,2,3
minimax/minimax-m2.7:high__json_schema PASS 100% 245 in → 337 out (325 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
minimax/minimax-m2.7:low PASS 100% 106 in → 139 out (132 reasoning)
1,2,3
minimax/minimax-m2.7:low__json_schema PASS 100% 245 in → 604 out (592 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
minimax/minimax-m2.7:medium PASS 100% 90 in → 87 out (79 reasoning)
1,2,3
minimax/minimax-m2.7:medium__json_schema PASS 100% 245 in → 401 out (389 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
minimax/minimax-m2.7:xhigh PASS 100% 111 in → 143 out (134 reasoning)
1,2,3
minimax/minimax-m3:low__json_schema PASS 100% 192 in → 18 out (13 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3"}
mistralai/devstral-2512 PASS 100% 72 in → 6 out
1,2,3
mistralai/devstral-2512__json_schema PASS 100% 72 in → 12 out
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
mistralai/mistral-medium-3-5:high PASS 100% 84 in → 251 out (238 reasoning)
1,2,3
mistralai/mistral-medium-3-5:high__json_schema PASS 100% 72 in → 110 out (90 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
mistralai/mistral-medium-3-5:low PASS 100% 84 in → 339 out (319 reasoning)
1,2,3
mistralai/mistral-medium-3-5:low__json_schema PASS 100% 72 in → 227 out (205 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
mistralai/mistral-medium-3-5:medium PASS 100% 84 in → 305 out (280 reasoning)
1,2,3
mistralai/mistral-medium-3-5:medium__json_schema PASS 100% 72 in → 137 out (128 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
mistralai/mistral-medium-3-5:none PASS 100% 84 in → 6 out (0 reasoning)
1,2,3
mistralai/mistral-medium-3-5:none__json_schema PASS 100% 72 in → 12 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
mistralai/mistral-medium-3-5:xhigh PASS 100% 84 in → 281 out (260 reasoning)
1,2,3
mistralai/mistral-medium-3-5:xhigh__json_schema PASS 100% 72 in → 146 out (131 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
moonshotai/kimi-k2.6:high PASS 100% 76 in → 131 out (124 reasoning)
1,2,3
moonshotai/kimi-k2.6:low PASS 100% 76 in → 98 out (91 reasoning)
1,2,3
moonshotai/kimi-k2.6:low__json_schema PASS 100% 75 in → 123 out (103 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
moonshotai/kimi-k2.6:medium PASS 100% 76 in → 109 out (102 reasoning)
1,2,3
moonshotai/kimi-k2.6:xhigh PASS 100% 76 in → 159 out (155 reasoning)
1,2,3
moonshotai/kimi-k2.6:xhigh__json_schema PASS 100% 76 in → 15 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
moonshotai/kimi-k2.7-code:high PASS 100% 76 in → 99 out (92 reasoning)
1,2,3
moonshotai/kimi-k2.7-code:low PASS 100% 76 in → 72 out (65 reasoning)
1,2,3
moonshotai/kimi-k2.7-code:low__json_schema PASS 100% 166 in → 96 out (84 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
moonshotai/kimi-k2.7-code:medium PASS 100% 76 in → 40 out (33 reasoning)
1,2,3
moonshotai/kimi-k2.7-code:medium__json_schema PASS 100% 76 in → 63 out (50 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
moonshotai/kimi-k2.7-code:xhigh PASS 100% 76 in → 78 out (71 reasoning)
1,2,3
moonshotai/kimi-k2.7-code:xhigh__json_schema PASS 100% 166 in → 52 out (40 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
moonshotai/kimi-k3:high PASS 100% 153 in → 114 out (94 reasoning)
1,2,3
moonshotai/kimi-k3:high__json_schema PASS 100% 153 in → 84 out (64 reasoning)
1,2,3
moonshotai/kimi-k3:low PASS 100% 153 in → 55 out (35 reasoning)
1,2,3
moonshotai/kimi-k3:low__json_schema PASS 100% 256 in → 104 out (79 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
moonshotai/kimi-k3:max PASS 100% 153 in → 83 out (63 reasoning)
1,2,3
moonshotai/kimi-k3:max__json_schema PASS 100% 153 in → 82 out (62 reasoning)
1,2,3
moonshotai/kimi-k3:medium PASS 100% 153 in → 110 out (90 reasoning)
1,2,3
moonshotai/kimi-k3:medium__json_schema PASS 100% 153 in → 97 out (77 reasoning)
1,2,3
moonshotai/kimi-k3:xhigh PASS 100% 153 in → 98 out (78 reasoning)
1,2,3
moonshotai/kimi-k3:xhigh__json_schema PASS 100% 256 in → 79 out (54 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 85 in → 113 out (101 reasoning)
1,2,3
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 85 in → 60 out (40 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-nano-30b-a3b:low PASS 100% 85 in → 256 out (261 reasoning)
1,2,3
nvidia/nemotron-3-nano-30b-a3b:low__json_schema PASS 100% 85 in → 183 out (160 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-nano-30b-a3b:medium PASS 100% 85 in → 263 out (227 reasoning)
1,2,3
nvidia/nemotron-3-nano-30b-a3b:medium__json_schema PASS 100% 85 in → 61 out (42 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 85 in → 6 out (0 reasoning)
1,2,3
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 85 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-nano-30b-a3b:xhigh PASS 100% 85 in → 158 out (158 reasoning)
1,2,3
nvidia/nemotron-3-nano-30b-a3b:xhigh__json_schema PASS 100% 85 in → 113 out (96 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-super-120b-a12b:high PASS 100% 85 in → 52 out (43 reasoning)
1,2,3
nvidia/nemotron-3-super-120b-a12b:high__json_schema PASS 100% 85 in → 83 out (67 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-super-120b-a12b:low PASS 100% 85 in → 124 out (114 reasoning)
1,2,3
nvidia/nemotron-3-super-120b-a12b:low__json_schema PASS 100% 85 in → 107 out (91 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-super-120b-a12b:medium PASS 100% 85 in → 46 out (33 reasoning)
1,2,3
nvidia/nemotron-3-super-120b-a12b:medium__json_schema PASS 100% 85 in → 58 out (42 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-super-120b-a12b:none PASS 100% 85 in → 6 out (0 reasoning)
1,2,3
nvidia/nemotron-3-super-120b-a12b:none__json_schema PASS 100% 85 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
nvidia/nemotron-3-super-120b-a12b:xhigh PASS 100% 85 in → 43 out (30 reasoning)
1,2,3
nvidia/nemotron-3-super-120b-a12b:xhigh__json_schema PASS 100% 85 in → 94 out (78 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-5.4-mini:high PASS 100% 74 in → 47 out (36 reasoning)
1,2,3
openai/gpt-5.4-mini:high__json_schema PASS 100% 115 in → 87 out (67 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-mini:low PASS 100% 74 in → 21 out (10 reasoning)
1,2,3
openai/gpt-5.4-mini:low__json_schema PASS 100% 115 in → 30 out (10 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-mini:medium PASS 100% 74 in → 24 out (13 reasoning)
1,2,3
openai/gpt-5.4-mini:medium__json_schema PASS 100% 115 in → 51 out (31 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-mini:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4-mini:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-mini:xhigh PASS 100% 74 in → 58 out (47 reasoning)
1,2,3
openai/gpt-5.4-mini:xhigh__json_schema PASS 100% 115 in → 99 out (79 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-nano:high__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-nano:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4-nano:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-nano:medium PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4-nano:medium__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-nano:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4-nano:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4-nano:xhigh PASS 100% 74 in → 39 out (28 reasoning)
1,2,3
openai/gpt-5.4-nano:xhigh__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4:high PASS 100% 74 in → 61 out (50 reasoning)
1,2,3
openai/gpt-5.4:high__json_schema PASS 100% 115 in → 329 out (309 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4:medium PASS 100% 74 in → 38 out (27 reasoning)
1,2,3
openai/gpt-5.4:medium__json_schema PASS 100% 115 in → 53 out (33 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.4:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.4:xhigh PASS 100% 74 in → 72 out (61 reasoning)
1,2,3
openai/gpt-5.4:xhigh__json_schema PASS 100% 115 in → 845 out (825 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.5:high PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.5:high__json_schema PASS 100% 115 in → 101 out (81 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.5:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.5:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.5:medium PASS 100% 74 in → 23 out (12 reasoning)
1,2,3
openai/gpt-5.5:medium__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.5:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.5:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.5:xhigh PASS 100% 74 in → 56 out (45 reasoning)
1,2,3
openai/gpt-5.5:xhigh__json_schema PASS 100% 115 in → 536 out (516 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:high PASS 100% 74 in → 22 out (11 reasoning)
1,2,3
openai/gpt-5.6-luna:high__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-luna:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:max PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-luna:max__json_schema PASS 100% 115 in → 97 out (77 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:medium PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-luna:medium__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-luna:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-luna:xhigh PASS 100% 74 in → 30 out (19 reasoning)
1,2,3
openai/gpt-5.6-luna:xhigh__json_schema PASS 100% 115 in → 74 out (54 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:high PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:high__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:max PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:max__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:medium PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:medium__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-sol:xhigh PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-sol:xhigh__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:high PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:high__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:low PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:low__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:max PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:max__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:medium PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:medium__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:none PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:none__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-5.6-terra:xhigh PASS 100% 74 in → 9 out (0 reasoning)
1,2,3
openai/gpt-5.6-terra:xhigh__json_schema PASS 100% 115 in → 18 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
openai/gpt-oss-120b:high PASS 100% 135 in → 121 out (102 reasoning)
1,2,3
openai/gpt-oss-120b:high__json_schema PASS 100% 137 in → 150 out (123 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-oss-120b:low PASS 100% 137 in → 41 out (20 reasoning)
1,2,3
openai/gpt-oss-120b:low__json_schema PASS 100% 133 in → 48 out (19 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-oss-120b:medium PASS 100% 133 in → 64 out (47 reasoning)
1,2,3
openai/gpt-oss-120b:medium__json_schema PASS 100% 135 in → 98 out (73 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-oss-120b:xhigh PASS 100% 135 in → 80 out (65 reasoning)
1,2,3
openai/gpt-oss-120b:xhigh__json_schema PASS 100% 135 in → 226 out (203 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-oss-20b:high PASS 100% 133 in → 341 out (328 reasoning)
1,2,3
openai/gpt-oss-20b:high__json_schema PASS 100% 122 in → 325 out (298 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
openai/gpt-oss-20b:low PASS 100% 137 in → 46 out (22 reasoning)
1,2,3
openai/gpt-oss-20b:medium PASS 100% 135 in → 105 out (90 reasoning)
1,2,3
openai/gpt-oss-20b:medium__json_schema PASS 100% 135 in → 134 out (112 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3"}
openai/gpt-oss-20b:xhigh PASS 100% 135 in → 249 out (220 reasoning)
1,2,3
openai/gpt-oss-20b:xhigh__json_schema PASS 100% 137 in → 161 out (134 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3"}
poolside/laguna-m.1:high PASS 100% 79 in → 271 out (263 reasoning)
1,2,3
poolside/laguna-m.1:high__json_schema PASS 100% 79 in → 258 out (240 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-m.1:low PASS 100% 79 in → 246 out (238 reasoning)
1,2,3
poolside/laguna-m.1:low__json_schema PASS 100% 79 in → 226 out (209 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-m.1:medium PASS 100% 79 in → 184 out (176 reasoning)
1,2,3
poolside/laguna-m.1:medium__json_schema PASS 100% 79 in → 137 out (119 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-m.1:none PASS 100% 79 in → 7 out (0 reasoning)
1,2,3
poolside/laguna-m.1:none__json_schema PASS 100% 79 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3" }
poolside/laguna-m.1:xhigh PASS 100% 79 in → 193 out (185 reasoning)
1,2,3
poolside/laguna-m.1:xhigh__json_schema PASS 100% 79 in → 175 out (162 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs-2.1:high PASS 100% 79 in → 134 out (128 reasoning)
1,2,3
poolside/laguna-xs-2.1:high__json_schema PASS 100% 79 in → 231 out (213 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-xs-2.1:low PASS 100% 79 in → 275 out (269 reasoning)
1,2,3
poolside/laguna-xs-2.1:low__json_schema PASS 100% 79 in → 304 out (287 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-xs-2.1:medium PASS 100% 79 in → 170 out (164 reasoning)
1,2,3
poolside/laguna-xs-2.1:medium__json_schema PASS 100% 79 in → 176 out (163 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs-2.1:none PASS 100% 79 in → 5 out (0 reasoning)
1,2,3
poolside/laguna-xs-2.1:none__json_schema PASS 100% 79 in → 17 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-xs-2.1:xhigh PASS 100% 79 in → 168 out (162 reasoning)
1,2,3
poolside/laguna-xs-2.1:xhigh__json_schema PASS 100% 79 in → 200 out (187 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs.2:high PASS 100% 116 in → 116 out (108 reasoning)
1,2,3
poolside/laguna-xs.2:high__json_schema PASS 100% 116 in → 82 out (69 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs.2:low PASS 100% 116 in → 113 out (105 reasoning)
1,2,3
poolside/laguna-xs.2:low__json_schema PASS 100% 116 in → 80 out (66 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs.2:medium PASS 100% 116 in → 57 out (49 reasoning)
1,2,3
poolside/laguna-xs.2:medium__json_schema PASS 100% 116 in → 130 out (117 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
poolside/laguna-xs.2:none PASS 100% 116 in → 7 out (0 reasoning)
1,2,3
poolside/laguna-xs.2:none__json_schema PASS 100% 116 in → 17 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
poolside/laguna-xs.2:xhigh PASS 100% 116 in → 136 out (128 reasoning)
1,2,3
poolside/laguna-xs.2:xhigh__json_schema PASS 100% 116 in → 103 out (89 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
qwen/qwen3.6-27b:high__json_schema PASS 100% 78 in → 814 out (666 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-27b:low PASS 100% 78 in → 293 out (251 reasoning)
1,2,3
qwen/qwen3.6-27b:medium PASS 100% 78 in → 811 out (664 reasoning)
1,2,3
qwen/qwen3.6-27b:none PASS 100% 80 in → 6 out (0 reasoning)
1,2,3
qwen/qwen3.6-27b:none__json_schema PASS 100% 80 in → 16 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-35b-a3b:high PASS 100% 78 in → 462 out (406 reasoning)
1,2,3
qwen/qwen3.6-35b-a3b:high__json_schema PASS 100% 78 in → 294 out (251 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-35b-a3b:low__json_schema PASS 100% 78 in → 651 out (542 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-35b-a3b:medium PASS 100% 78 in → 277 out (240 reasoning)
1,2,3
qwen/qwen3.6-35b-a3b:medium__json_schema PASS 100% 78 in → 536 out (450 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-35b-a3b:xhigh PASS 100% 78 in → 329 out (287 reasoning)
1,2,3
qwen/qwen3.6-35b-a3b:xhigh__json_schema PASS 100% 78 in → 381 out (313 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
qwen/qwen3.6-flash:low PASS 100% 78 in → 344 out (334 reasoning)
1,2,3
qwen/qwen3.7-flash:medium PASS 100% 78 in → 746 out (736 reasoning)
1,2,3
qwen/qwen3.7-flash:none PASS 100% 80 in → 5 out (0 reasoning)
1,2,3
qwen/qwen3.7-flash:xhigh PASS 100% 78 in → 343 out (333 reasoning)
1,2,3
qwen/qwen3.7-max:high PASS 100% 78 in → 490 out (480 reasoning)
1,2,3
qwen/qwen3.7-max:low PASS 100% 78 in → 491 out (481 reasoning)
1,2,3
qwen/qwen3.7-max:medium PASS 100% 78 in → 349 out (339 reasoning)
1,2,3
qwen/qwen3.7-max:none PASS 100% 80 in → 5 out (0 reasoning)
1,2,3
qwen/qwen3.7-max:xhigh PASS 100% 78 in → 271 out (261 reasoning)
1,2,3
qwen/qwen3.7-plus:high PASS 100% 78 in → 436 out (426 reasoning)
1,2,3
qwen/qwen3.7-plus:low PASS 100% 78 in → 645 out (635 reasoning)
1,2,3
qwen/qwen3.7-plus:medium PASS 100% 78 in → 354 out (344 reasoning)
1,2,3
qwen/qwen3.7-plus:none PASS 100% 80 in → 5 out (0 reasoning)
1,2,3
qwen/qwen3.7-plus:xhigh PASS 100% 78 in → 367 out (357 reasoning)
1,2,3
tencent/hy3:high PASS 100% 80 in → 172 out (165 reasoning)
1,2,3
tencent/hy3:high__json_schema PASS 100% 80 in → 204 out (157 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
tencent/hy3:low PASS 100% 80 in → 96 out (89 reasoning)
1,2,3
tencent/hy3:low__json_schema PASS 100% 80 in → 132 out (104 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
tencent/hy3:medium PASS 100% 80 in → 145 out (128 reasoning)
1,2,3
tencent/hy3:medium__json_schema PASS 100% 80 in → 207 out (175 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
tencent/hy3:none PASS 100% 83 in → 6 out (0 reasoning)
1,2,3
tencent/hy3:none__json_schema PASS 100% 83 in → 12 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
tencent/hy3:xhigh PASS 100% 80 in → 299 out (292 reasoning)
1,2,3
tencent/hy3:xhigh__json_schema PASS 100% 80 in → 278 out (224 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling-small:high PASS 100% 83 in → 98 out (82 reasoning)
1,2,3
thinkingmachines/inkling-small:high__json_schema PASS 100% 83 in → 112 out (94 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling-small:low PASS 100% 83 in → 23 out (12 reasoning)
1,2,3
thinkingmachines/inkling-small:low__json_schema PASS 100% 83 in → 30 out (12 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3"}
thinkingmachines/inkling-small:max PASS 100% 83 in → 75 out (64 reasoning)
1,2,3
thinkingmachines/inkling-small:max__json_schema PASS 100% 83 in → 121 out (103 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling-small:medium PASS 100% 83 in → 34 out (23 reasoning)
1,2,3
thinkingmachines/inkling-small:medium__json_schema PASS 100% 83 in → 109 out (91 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling-small:xhigh PASS 100% 83 in → 64 out (53 reasoning)
1,2,3
thinkingmachines/inkling:high PASS 100% 83 in → 115 out (97 reasoning)
1,2,3
thinkingmachines/inkling:high__json_schema PASS 100% 83 in → 65 out (49 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
thinkingmachines/inkling:low PASS 100% 83 in → 24 out (12 reasoning)
1,2,3
thinkingmachines/inkling:low__json_schema PASS 100% 83 in → 31 out (12 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling:max PASS 100% 83 in → 61 out (49 reasoning)
1,2,3
thinkingmachines/inkling:max__json_schema PASS 100% 83 in → 125 out (107 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling:medium PASS 100% 83 in → 58 out (46 reasoning)
1,2,3
thinkingmachines/inkling:medium__json_schema PASS 100% 83 in → 67 out (49 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
thinkingmachines/inkling:none PASS 100% 81 in → 9 out (0 reasoning)
1,2,3
thinkingmachines/inkling:xhigh PASS 100% 83 in → 118 out (106 reasoning)
1,2,3
x-ai/grok-4.3:high PASS 100% 253 in → 279 out (274 reasoning)
1,2,3
x-ai/grok-4.3:high__json_schema PASS 100% 316 in → 330 out (320 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.3:low PASS 100% 259 in → 139 out (134 reasoning)
1,2,3
x-ai/grok-4.3:low__json_schema PASS 100% 322 in → 311 out (301 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.3:max PASS 100% 253 in → 159 out (154 reasoning)
1,2,3
x-ai/grok-4.3:max__json_schema PASS 100% 316 in → 293 out (283 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.3:medium__json_schema PASS 100% 322 in → 405 out (391 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
x-ai/grok-4.3:none PASS 100% 251 in → 5 out (0 reasoning)
1,2,3
x-ai/grok-4.3:none__json_schema PASS 100% 314 in → 10 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.3:xhigh PASS 100% 253 in → 157 out (152 reasoning)
1,2,3
x-ai/grok-4.3:xhigh__json_schema PASS 100% 316 in → 280 out (270 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.5:high PASS 100% 278 in → 109 out (104 reasoning)
1,2,3
x-ai/grok-4.5:high__json_schema PASS 100% 354 in → 319 out (307 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers":"1,2,3"}
x-ai/grok-4.5:low PASS 100% 278 in → 105 out (100 reasoning)
1,2,3
x-ai/grok-4.5:low__json_schema PASS 100% 354 in → 230 out (218 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
x-ai/grok-4.5:max PASS 100% 278 in → 103 out (98 reasoning)
1,2,3
x-ai/grok-4.5:max__json_schema PASS 100% 354 in → 264 out (252 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
x-ai/grok-4.5:medium PASS 100% 278 in → 84 out (79 reasoning)
1,2,3
x-ai/grok-4.5:medium__json_schema PASS 100% 354 in → 263 out (251 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
x-ai/grok-4.5:xhigh PASS 100% 278 in → 181 out (176 reasoning)
1,2,3
x-ai/grok-4.5:xhigh__json_schema PASS 100% 354 in → 237 out (225 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
z-ai/glm-4.7-flash:high PASS 100% 73 in → 338 out (305 reasoning)
1,2,3
z-ai/glm-4.7-flash:none PASS 100% 73 in → 5 out (0 reasoning)
1,2,3
z-ai/glm-4.7-flash:none__json_schema PASS 100% 73 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
z-ai/glm-5.2:high PASS 100% 80 in → 271 out (264 reasoning)
1,2,3
z-ai/glm-5.2:low__json_schema PASS 100% 80 in → 301 out (234 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
z-ai/glm-5.2:medium PASS 100% 80 in → 253 out (215 reasoning)
1,2,3
z-ai/glm-5.2:medium__json_schema PASS 100% 262 in → 112 out (99 reasoning)
1,2,3
JSON Schema Structured Output
(raw) {"line_numbers": "1,2,3"}
z-ai/glm-5.2:none__json_schema PASS 100% 74 in → 14 out (0 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
z-ai/glm-5.2:xhigh PASS 100% 80 in → 378 out (371 reasoning)
1,2,3
z-ai/glm-5.2:xhigh__json_schema PASS 100% 80 in → 268 out (209 reasoning)
1,2,3
JSON Schema Structured Output
(raw) { "line_numbers": "1,2,3" }
anthropic/claude-fable-5:high FAIL 0% 103 in → 23 out (13 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:high__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:low FAIL 0% 103 in → 16 out (14 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:low__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:max__json_schema FAIL 0% 334 in → 74 out (27 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:medium FAIL 0% 103 in → 23 out (13 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:medium__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:xhigh FAIL 0% 103 in → 23 out (13 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-fable-5:xhigh__json_schema FAIL 0% 334 in → 34 out (13 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:high FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:high__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:low FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:low__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:max FAIL 0% 103 in → 121 out (38 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:max__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:medium FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:medium__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:none FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:none__json_schema FAIL 0% 334 in → 19 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:xhigh FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-opus-4.8:xhigh__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:high FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:high__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:low FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:medium FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:medium__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:none FAIL 0% 103 in → 9 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
anthropic/claude-sonnet-5:xhigh__json_schema FAIL 0% 334 in → 20 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-pro:none__json_schema FAIL 0% 74 in → 13 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
google/gemini-3.1-pro-preview:high__json_schema FAIL 0% 143 in → 282 out (270 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers":"1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
google/gemini-3.1-pro-preview:medium__json_schema FAIL 0% 143 in → 160 out (147 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
google/gemini-3.6-flash:max__json_schema FAIL 0% 143 in → 420 out (407 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers":"1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
google/gemma-4-26b-a4b-it:low FAIL 0% 92 in → 256 out (187 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
google/gemma-4-31b-it:high__json_schema FAIL 0% 91 in → 146 out (87 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
liquid/lfm-2-24b-a2b FAIL 0% 80 in → 2 out
1
Failure: Expected '1,2,3', got '1'
Invalid structured output. Output: 1
JSON Schema Structured Output
Structured Output Error
Structured output schema validation failed: $ must be of type object
Failure: Structured output schema validation failed: $ must be of type object
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
minimax/minimax-m3:high FAIL 0% 244 in → 72 out (53 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:high__json_schema FAIL 0% 231 in → 16 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:low FAIL 0% 244 in → 57 out (47 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:medium FAIL 0% 244 in → 48 out (37 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:medium__json_schema FAIL 0% 231 in → 16 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:xhigh FAIL 0% 244 in → 88 out (73 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
minimax/minimax-m3:xhigh__json_schema FAIL 0% 231 in → 16 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
Invalid JSON. Output: 1,2,3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 3 (char 2)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
moonshotai/kimi-k2.7-code:high__json_schema FAIL 0% 76 in → 64 out (49 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) {"line_numbers": "1, 2, 3"}
Failure: Expected '1,2,3', got '1, 2, 3'
openai/gpt-5.4-nano:high FAIL 0% 74 in → 11 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
openai/gpt-oss-20b:low__json_schema FAIL 0% 133 in → 12,662 out (8 reasoning)
(empty output)
JSON Schema Structured Output
(raw) { "line_numbers" : "" }
Failure: Expected '1,2,3', got ''
qwen/qwen3.6-27b:high FAIL 0% 78 in → 1,066 out (1 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-27b:low__json_schema FAIL 0% 78 in → 799 out (602 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-27b:medium__json_schema FAIL 0% 78 in → 958 out (1 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-27b:xhigh FAIL 0% 78 in → 770 out (590 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-27b:xhigh__json_schema FAIL 0% 78 in → 1,247 out (900 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-35b-a3b:low FAIL 0% 78 in → 1,485 out (1,473 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-35b-a3b:none FAIL 0% 80 in → 8 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-35b-a3b:none__json_schema FAIL 0% 80 in → 19 out (0 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
qwen/qwen3.6-flash:high FAIL 0% 78 in → 1,414 out (1,402 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b0decdb4-7b3d-93a0-9358-ff80ab31fd6a","request_id":"b0decdb4-7b3d-93a0-9358-ff80ab31fd6a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5f4f1934-bd65-9951-ade7-92690335c3b7","request_id":"5f4f1934-bd65-9951-ade7-92690335c3b7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:medium FAIL 0% 78 in → 1,043 out (1,031 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-31899a3d-9488-90b4-91cd-d25f0cac5eaa","request_id":"31899a3d-9488-90b4-91cd-d25f0cac5eaa"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:none FAIL 0% 80 in → 7 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-5157e0e2-7c8b-94d4-a099-6f52399e1416","request_id":"5157e0e2-7c8b-94d4-a099-6f52399e1416"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.6-flash:xhigh FAIL 0% 78 in → 1,657 out (1,645 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-ecfa59ef-27d9-993a-9e22-4e4ba9fc7652","request_id":"ecfa59ef-27d9-993a-9e22-4e4ba9fc7652"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:high FAIL 0% 78 in → 286 out (274 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-5e07087d-3e6b-915d-a469-c567fae7e0bf"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:low FAIL 0% 78 in → 1,159 out (1,147 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-bff82b3b-951f-96b3-a185-0e24a056c078"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
qwen/qwen3.7-flash:max FAIL 0% 78 in → 1,929 out (1,917 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-78d218f6-61a8-9ed9-b362-9a4d2bbee969"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-80a2cf7d-d013-9d22-962c-03689e9a1400"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-43523f37-0cc8-9b58-aab8-063a5afa0385"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': 'data: {"error":{"code":"invalid_parameter_error","param":null,"message":"\'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error"},"id":"chatcmpl-c32eff8c-9ec8-9fbf-bf50-8b7aa75fb78f"}\n\n', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-66e0c0e5-c4c2-9181-982f-c30eaeece846","request_id":"66e0c0e5-c4c2-9181-982f-c30eaeece846"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-9cd0907b-574b-9904-b0a2-1100182b2f9f","request_id":"9cd0907b-574b-9904-b0a2-1100182b2f9f"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-36f18bbd-f6a4-90dc-81d7-0b5cd3b23a9b","request_id":"36f18bbd-f6a4-90dc-81d7-0b5cd3b23a9b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-fe35b244-f250-9d9c-abca-11d7915d3cf1","request_id":"fe35b244-f250-9d9c-abca-11d7915d3cf1"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-b07f95cc-6fbd-9f92-ba76-843952a085e7","request_id":"b07f95cc-6fbd-9f92-ba76-843952a085e7"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-c2fb0e80-f81e-9f94-aa8e-4515152f7d6a","request_id":"c2fb0e80-f81e-9f94-aa8e-4515152f7d6a"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-96f4e62c-6065-91c8-ae2a-3ec6c5f0fed8","request_id":"96f4e62c-6065-91c8-ae2a-3ec6c5f0fed8"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-25c748c9-ceb4-92d4-9c80-ac4a3933a905","request_id":"25c748c9-ceb4-92d4-9c80-ac4a3933a905"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-a6e8577c-f8c3-9e0d-90a1-9f891ba1771d","request_id":"a6e8577c-f8c3-9e0d-90a1-9f891ba1771d"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
(empty output)
JSON Schema Structured Output
Failure: Error code: 400 - {'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"<400> InternalError.Algo.InvalidParameter: \'messages\' must contain the word \'json\' in some form, to use \'response_format\' of type \'json_object\'.","type":"invalid_request_error","param":null,"code":"invalid_parameter_error"},"id":"chatcmpl-f8f569a3-2a59-9b2c-b2c8-20aff6ab681b","request_id":"f8f569a3-2a59-9b2c-b2c8-20aff6ab681b"}', 'provider_name': 'Alibaba', 'is_byok': False}}, 'user_id': 'user_3FrVBkmFqVIuZ58dEh7AU8vhXUV'}
thinkingmachines/inkling-small:none FAIL 0% 83 in → 8 out (0 reasoning)
1,2,3<|end_message|>
Failure: Expected '1,2,3', got '1,2,3<|end_message|>'
Invalid JSON. Output: 1,2,3{"line_numbers": "1,2,3"}
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
(empty output)
JSON Schema Structured Output
Failure: OpenAI-compatible response contained provider error: json_schema response format is not supported for model: thinkingmachines/Inkling-Small
Invalid JSON. Output: 1,2,3{ "line_numbers": "1,2,3" }
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Invalid JSON. Output: 1,2,3
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
Failure: Failed to parse structured JSON response: Extra data: line 1 column 2 (char 1)
x-ai/grok-4.3:medium FAIL 0% 259 in → 244 out (164 reasoning)
1,2,3 The provided git grep -n output directly lists three matches for the string "import" (each on its own line) inside src/app.py. The only numeric values appearing after the filename prefix are therefore the line numbers 1, 2 and 3; extracting those numbers and emitting them as a comma-separated list satisfies the required output format with no other text.
Failure: Expected '1,2,3', got '1,2,3 The provided git grep -n output directly lists three matches for the string "import" (each on its own line) inside src/app.py. The only numeric values appearing after the filename prefix are therefore the line numbers 1, 2 and 3; extracting those numbers and emitting them as a comma-separated list satisfies the required output format with no other text.'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:low FAIL 0% 73 in → 304 out (261 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
z-ai/glm-4.7-flash:low__json_schema FAIL 0% 73 in → 295 out (251 reasoning)
1, 2, 3
JSON Schema Structured Output
(raw) { "line_numbers": "1, 2, 3" }
Failure: Expected '1,2,3', got '1, 2, 3'
z-ai/glm-4.7-flash:medium FAIL 0% 73 in → 291 out (251 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-4.7-flash:xhigh FAIL 0% 73 in → 298 out (265 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Invalid JSON. Output:
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
z-ai/glm-5.2:low FAIL 0% 80 in → 386 out (277 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'
z-ai/glm-5.2:none FAIL 0% 74 in → 8 out (0 reasoning)
1, 2, 3
Failure: Expected '1,2,3', got '1, 2, 3'