# AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development

arXiv: https://arxiv.org/abs/2601.13597 · HTML: https://mlsanity.com/paper/2601.13597/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://mlsanity.com/paper/2507.09089/) (2025)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows](https://mlsanity.com/paper/2507.08149/) (2025)
- [Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects](https://mlsanity.com/paper/2406.17910/) (2024)
- [Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise](https://mlsanity.com/paper/2412.06603/) (2024)
- ["It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Tools](https://mlsanity.com/paper/2212.03491/) (2022)
- [The Role of Generative AI in Software Development Productivity: A Pilot Case Study](https://mlsanity.com/paper/2406.00560/) (2024)
- [Agentic AI for Software: thoughts from Software Engineering community](https://mlsanity.com/paper/2508.17343/) (2025)
- [The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks](https://mlsanity.com/paper/2506.10051/) (2025)
- [Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants](https://mlsanity.com/paper/2402.08431/) (2024)
- [Intuition to Evidence: Measuring AI's True Impact on Developer Productivity](https://mlsanity.com/paper/2509.19708/) (2025)
- [Impact of AI-tooling on the Engineering Workspace](https://mlsanity.com/paper/2406.07683/) (2024)
- [Harnessing the Potential of Gen-AI Coding Assistants in Public Sector Software Development](https://mlsanity.com/paper/2409.17434/) (2024)
- [Self-Admitted GenAI Usage in Open-Source Software](https://mlsanity.com/paper/2507.10422/) (2025)
- [Ten Simple Rules for AI-Assisted Coding in Science](https://mlsanity.com/paper/2510.22254/) (2025)
- [Quality evaluation of Tabby coding assistant using real source code snippets](https://mlsanity.com/paper/2504.08650/) (2025)
- [Creating benchmarkable components to measure the quality ofAI-enhanced developer tools](https://mlsanity.com/paper/2504.12211/) (2025)
- [On the Future of Software Reuse in the Era of AI Native Software Engineering](https://mlsanity.com/paper/2508.19834/) (2025)

## Built on by (most cited first, 11 of 11)

- [VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation](https://mlsanity.com/paper/2605.08553/) (2026)
- [Collaborator or Assistant? How AI Coding Agents Partition Work Across Pull Request Lifecycles](https://mlsanity.com/paper/2605.08017/) (2026)
- [Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineering](https://mlsanity.com/paper/2607.01087/) (2026)
- [Mining Architectural Quality Under Agentic AI Adoption: A Causal Study of Java Repositories](https://mlsanity.com/paper/2606.13298/) (2026)
- [Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI](https://mlsanity.com/paper/2607.01418/) (2026)
- [Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption](https://mlsanity.com/paper/2606.26289/) (2026)
- [From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software](https://mlsanity.com/paper/2607.05677/) (2026)
- [Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code](https://mlsanity.com/paper/2607.09902/) (2026)
- [Case study: solving P-99 with LPTP and an LLM](https://mlsanity.com/paper/2607.21196/) (2026)
- [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026)
- [A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption](https://mlsanity.com/paper/2608.25241/) (2026)

## Abstract (arXiv)

Large language model (LLM) based coding agents increasingly act as autonomous contributors that generate and merge pull requests, yet their real-world effects on software projects are unclear-especially compared with widely adopted IDE-based AI assistants. We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls. Using the AIDev dataset, we define adoption as the first agent-generated pull request and analyze monthly repository-level outcomes spanning development velocity (commits, lines added) and software quality (static-analysis warnings, cognitive complexity, duplication, and comment density). Results show large, front-loaded velocity gains only when agents are the first observable AI tool in a project; repositories with prior AI IDE usage experience minimal or short-lived throughput increases. In contrast, quality risks are persistent across settings, with static-analysis warnings and cognitive complexity rising by roughly 18% and 39%, indicating sustained agent-induced technical debt even when velocity advantages fade. These heterogeneous effects suggest diminishing returns to AI assistance and highlight the need for quality safeguards, provenance tracking, and selective deployment of autonomous agents. Our findings establish an empirical basis for understanding how agentic and IDE-based tools interact, and motivate research on balancing acceleration with maintainability in AI-integrated development workflows. The replication package for this study is publicly available at https://github.com/shyamagarwal13/agentic-coding-impact.