# Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions

arXiv: https://arxiv.org/abs/2604.00436 · HTML: https://mlsanity.com/paper/2604.00436/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2026.

## Built on by (most cited first, 14 of 14)

- [TraceLab: Characterizing Coding Agent Workloads for LLM Serving](https://mlsanity.com/paper/2606.30560/) (2026)
- [How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions](https://mlsanity.com/paper/2605.29442/) (2026)
- [An Empirical Study of Agent Skills for Healthcare: Practice, Gaps, and Governance](https://mlsanity.com/paper/2605.02709/) (2026)
- [An Empirical Study of Proactive Coding Assistants in Real-World Software Development](https://mlsanity.com/paper/2605.05700/) (2026)
- [Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study](https://mlsanity.com/paper/2606.12231/) (2026)
- [Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories](https://mlsanity.com/paper/2608.10319/) (2026)
- [From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software](https://mlsanity.com/paper/2607.05677/) (2026)
- [Bespoke Visual Assistance: What and How do Blind and Low-Vision People Create with Agentic Programming?](https://mlsanity.com/paper/2607.21760/) (2026)
- [Investigating Assistant Bias in LLM User Simulators Using a Role Vector](https://mlsanity.com/paper/2609.00608/) (2026)
- [The Work Behind Delegation: A Framework for Supervising AI Coding Agents](https://mlsanity.com/paper/2609.24234/) (2026)
- [Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase](https://mlsanity.com/paper/2609.29744/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)
- [ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding](https://mlsanity.com/paper/2610.08662/) (2026)
- [Coding-Agent Benchmarks Should Match Their Users' Task Flows](https://mlsanity.com/paper/2610.09633/) (2026)

## Abstract (arXiv)

IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.