# SWE-chat: Coding Agent Interactions From Real Users in the Wild

arXiv: https://arxiv.org/abs/2604.20779 · HTML: https://mlsanity.com/paper/2604.20779/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 80 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 32)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://mlsanity.com/paper/2510.04374/) (2025)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://mlsanity.com/paper/2312.04724/) (2023)
- [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://mlsanity.com/paper/2507.09089/) (2025)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [Copilot Arena: A Platform for Code LLM Evaluation in the Wild](https://mlsanity.com/paper/2502.09328/) (2025)
- [ProSec: Fortifying Code LLMs with Proactive Security Alignment](https://mlsanity.com/paper/2411.12882/) (2024)
- [Composer 2 Technical Report](https://mlsanity.com/paper/2603.24477/) (2026)
- [Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents](https://mlsanity.com/paper/2510.24702/) (2025)
- [Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction](https://mlsanity.com/paper/2512.04987/) (2025)
- [When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback](https://mlsanity.com/paper/2502.18413/) (2025)
- [SERA: Soft-Verified Efficient Repository Agents](https://mlsanity.com/paper/2601.20789/) (2026)
- [How Well Does Agent Development Reflect Real-World Work?](https://mlsanity.com/paper/2603.01203/) (2026)

## Built on by (most cited first, 30 of 43)

- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions](https://mlsanity.com/paper/2605.29442/) (2026)
- [Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents](https://mlsanity.com/paper/2605.12894/) (2026)
- [LLMs Get Lost in Evolving User Intent](https://mlsanity.com/paper/2607.20734/) (2026)
- [ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions](https://mlsanity.com/paper/2605.20087/) (2026)
- [Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents](https://mlsanity.com/paper/2606.13995/) (2026)
- [SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution](https://mlsanity.com/paper/2605.08366/) (2026)
- [Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents](https://mlsanity.com/paper/2609.12205/) (2026)
- [EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments](https://mlsanity.com/paper/2607.02440/) (2026)
- [RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions](https://mlsanity.com/paper/2605.01104/) (2026)
- [Offloading Score: Measuring AI Reliance Through Counterfactual Workflows](https://mlsanity.com/paper/2605.29392/) (2026)
- [CurateEvo: Data-Curation Evolving for Agentic Post-Training](https://mlsanity.com/paper/2607.06140/) (2026)
- [Align AI to Dynamic Human-AI Workflows](https://mlsanity.com/paper/2607.14240/) (2026)
- [Learning from 53.6K Real-World Developer Edits of AI-Generated Code](https://mlsanity.com/paper/2607.25130/) (2026)
- [Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories](https://mlsanity.com/paper/2608.10319/) (2026)
- [Beyond Code Generation: Reliability, Verification, and Cost Economics in the Agentic Software Development Lifecycle](https://mlsanity.com/paper/2609.04681/) (2026)
- [Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise](https://mlsanity.com/paper/2609.28919/) (2026)
- [LLM Anonymization Against Agentic Re-Identification](https://mlsanity.com/paper/2605.30848/) (2026)
- [Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention](https://mlsanity.com/paper/2608.03555/) (2026)
- [Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration](https://mlsanity.com/paper/2608.11460/) (2026)
- [From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation](https://mlsanity.com/paper/2608.20195/) (2026)
- [RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests](https://mlsanity.com/paper/2608.27831/) (2026)
- [PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems](https://mlsanity.com/paper/2608.28378/) (2026)
- [REVISE: Validity-Guided Recovery for Online Revisions in Agent Workflows](https://mlsanity.com/paper/2609.00643/) (2026)
- [Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks](https://mlsanity.com/paper/2609.17817/) (2026)
- [Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents](https://mlsanity.com/paper/2609.19587/) (2026)
- [The Work Behind Delegation: A Framework for Supervising AI Coding Agents](https://mlsanity.com/paper/2609.24234/) (2026)
- [Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase](https://mlsanity.com/paper/2609.29744/) (2026)

## Abstract (arXiv)

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains almost 18,000 sessions, comprising more than 229,000 user prompts and 2 million agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 25%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Only 59% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs - through corrections, failure reports, and interruptions - in 50% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.