# SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv: https://arxiv.org/abs/2606.30573 · HTML: https://mlsanity.com/paper/2606.30573/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 12 of 12)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios](https://mlsanity.com/paper/2512.18470/) (2025)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains](https://mlsanity.com/paper/2603.13686/) (2026)
- [HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?](https://mlsanity.com/paper/2604.09408/) (2026)
- [Agentic Rubrics as Contextual Verifiers for SWE Agents](https://mlsanity.com/paper/2601.04171/) (2026)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [LHAW: Controllable Underspecification for Long-Horizon Tasks](https://mlsanity.com/paper/2602.10525/) (2026)
- [SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement](https://mlsanity.com/paper/2509.18808/) (2025)
- [Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement](https://mlsanity.com/paper/2606.05920/) (2026)

## Built on by (most cited first, 11 of 11)

- [Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments](https://mlsanity.com/paper/2609.04148/) (2026)
- [SWE-Touch: Benchmarking Coding Agents When Users Touch the Code](https://mlsanity.com/paper/2608.02499/) (2026)
- [SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories](https://mlsanity.com/paper/2608.09799/) (2026)
- [Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories](https://mlsanity.com/paper/2608.10319/) (2026)
- [Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent](https://mlsanity.com/paper/2609.38604/) (2026)
- [RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests](https://mlsanity.com/paper/2608.27831/) (2026)
- [MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding](https://mlsanity.com/paper/2609.14992/) (2026)
- [Coding-Agent Benchmarks Should Match Their Users' Task Flows](https://mlsanity.com/paper/2610.09633/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)
- [StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents](https://mlsanity.com/paper/2610.17616/) (2026)
- [Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents](https://mlsanity.com/paper/2610.17757/) (2026)

## Abstract (arXiv)

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.