mlsanityData to 7 Oct 2026

Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

arXiv 2607.21674 ↗First posted 23 Jul 2026cs.SEcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Output format is not a neutral implementation detail -- it can reorder model rankings, amplify or suppress individual model differences, and determine whether a coding agent succeeds or fails. We conducted a controlled single-round experiment with 3 models (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) x 3 output formats (full file, JSON Patch, unified diff) x 6 tasks x 20 repetitions, totaling 4,013 runs across 4 open-source projects. Only one project (tqdm) yielded non-zero success rates: dotenv, requests, and jsoup yielded zero successes in 2,551 runs. Our central finding is a format x model interaction with no universally optimal format. Doubao achieves 94% success with JSON Patch (Cohen's h = 1.57, p < 0.001), DeepSeek excels at unified diff (66%, h = 0.63), and Qwen shows a small but significant full-file preference (50%, h = 0.29, p < 0.05). Beyond these headline results, we identify a distinct failure mechanism -- format misuse -- where agents correctly diagnose a problem but execute it with excessive scope, most vividly when a one-line fix is applied as a full-file replacement. We propose a model-specific output strategy, a tool-design principle that constrains format semantics to the agent's own localization step, and release all data and templates for reproducibility. All experimental data are available at https://doi.org/10.5281/zenodo.21505157.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 15 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

10 of 15 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Where LLM Agents Fail and How They can Learn From Failures29 Sep
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  3. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 6

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  4. 2023Efficient Guided Generation for Large Language Modelscited by 192
  5. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  6. 2026Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Architecture as Capability Equalizer for Coding Agents
  2. 2026When Agents Go Quiet: Output Generation Capacity and Format-Cost Separation for LLM Document Synthesis
  3. 2026Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses
  4. 2026Interaction Scaling: Grounding the Third Axis of Test-Time Compute
  5. 2026Capacity, Not Format: Rethinking Structured Reasoning Failures
  6. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Serving
  7. 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
  8. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality