SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 52 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
26 of 52 references.
Closest ancestors · 12
Same direction, newest first
- 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders23 Jul
- 2026Humans are Missing from AI Coding Agent Research4 Jul
- 2026SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions29 Jun
- 2026Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents12 Jun
- 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions22 May
- 2026Hedwig: Dynamic Autonomy for Coding Agents Under Local Oversight12 May
- 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks25 Mar
- 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios20 Dec
- 2025How can we assess human-agent interactions? Case studies in software agent design10 Oct
- 2025SWE-smith: Scaling Data for Software Engineering Agents30 Apr
- 2025SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering10 Feb
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 14
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2024GPT-4o System Cardcited by 4,467
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
- 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 57
- 2025Non-Collaborative User Simulators for Tool Agentscited by 17
- 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 16
- 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026SWE-Together: Evaluating Coding Agents in Interactive User Sessions
- 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositories
- 2026How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
- 2026Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
- 2026SWE-chat: Coding Agent Interactions From Real Users in the Wild
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
- 2025TOM-SWE: User Mental Modeling For Software Engineering Agents