# Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

arXiv: https://arxiv.org/abs/2606.13995 · HTML: https://mlsanity.com/paper/2606.13995/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 55 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 16 of 16)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents](https://mlsanity.com/paper/2511.03690/) (2025)
- [SWE-chat: Coding Agent Interactions From Real Users in the Wild](https://mlsanity.com/paper/2604.20779/) (2026)
- [Python Code Generation by Asking Clarification Questions](https://mlsanity.com/paper/2212.09885/) (2022)
- [When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback](https://mlsanity.com/paper/2502.18413/) (2025)
- [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://mlsanity.com/paper/2502.13791/) (2025)

## Built on by (most cited first, 7 of 7)

- [SWE-Touch: Benchmarking Coding Agents When Users Touch the Code](https://mlsanity.com/paper/2608.02499/) (2026)
- [A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents](https://mlsanity.com/paper/2608.09072/) (2026)
- [BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests](https://mlsanity.com/paper/2608.02685/) (2026)
- [RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests](https://mlsanity.com/paper/2608.27831/) (2026)
- [Coding-Agent Benchmarks Should Match Their Users' Task Flows](https://mlsanity.com/paper/2610.09633/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)
- [Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents](https://mlsanity.com/paper/2610.17757/) (2026)

## Abstract (arXiv)

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.