mlsanityData to 14 Oct 2026

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

arXiv 2507.09089 ↗First posted 12 Jul 2025cs.AI cs.HC cs.SEcited by 101

Not on the map. Why?

Cited by 43 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 12 Jul 2025.

Builds on

16 of 41 references.

References · 16

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  7. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  8. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  9. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  10. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  11. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  12. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
4 more
  1. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  2. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  3. 2024How much does AI impact development speed? An enterprise-based randomized controlled trialcited by 20
  4. 2024The Impact of Large Language Models on Open-source Innovation: Evidence from GitHub Copilotcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 86.

101 papers.

Sort:

By area:

  1. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 43
  2. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 38
  3. 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 32
  4. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 24
  5. 2026How AI Impacts Skill Formationcited by 24
  6. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  7. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 12
  8. 2026AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Developmentcited by 11
  9. 2025Revolution or Hype? Seeking the Limits of Large Models in Hardware Designcited by 9
  10. 2026Measuring AI R&D Automationcited by 9
  11. 2025Understanding, Protecting, and Augmenting Human Cognition with Generative AI: A Synthesis of the CHI 2025 Tools for Thought Workshopcited by 7
  12. 2025Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Developmentcited by 7
38 more
  1. 2026More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requestscited by 7
  2. 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiversecited by 7
  3. 2026Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activitycited by 6
  4. 2026Are Coding Agents Generating Over-Mocked Tests? An Empirical Studycited by 6
  5. 2026Measuring Mid-2025 LLM-Assistance on Novice Performance in Biologycited by 6
  6. 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generationcited by 6
  7. 2025Intuition to Evidence: Measuring AI's True Impact on Developer Productivitycited by 5
  8. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Timecited by 5
  9. 2025Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaborationcited by 4
  10. 2026Knowledge Activation: AI Skills as the Institutional Knowledge Primitive for Agentic Software Developmentcited by 4
  11. 2026Life After Benchmark Saturation: A Case Study of CORE-Benchcited by 4
  12. 2026Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptancecited by 3
  13. 2026The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Developmentcited by 3
  14. 2026Steering LLMs via Scalable Interactive Oversightcited by 2
  15. 2026Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impactcited by 2
  16. 2026Orchestrating Human-AI Software Delivery: A Retrospective Longitudinal Field Study of Three Software Modernization Programscited by 2
  17. 2026More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processescited by 2
  18. 2026The Semi-Executable Stack: Agentic Software Engineering and the Expanding Scope of SEcited by 2
  19. 2026To What Extent Does Agent-generated Code Require Maintenance? An Empirical Studycited by 2
  20. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  21. 2026Human-AI Productivity Paradoxes: Modeling the Interplay of Skill, Effort, and AI Assistancecited by 2
  22. 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Developmentcited by 2
  23. 2026Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational studycited by 2
  24. 2026How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Studycited by 2
  25. 2026(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understandingcited by 2
  26. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challengescited by 1
  27. 2025Collaborating with GenAI: Incentives and Replacementscited by 1
  28. 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
  29. 2025Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planningcited by 1
  30. 2026The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebasecited by 1
  31. 2026Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projectscited by 1
  32. 2026More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystemscited by 1
  33. 2026Offloading Score: Measuring AI Reliance Through Counterfactual Workflowscited by 1
  34. 2026How AI Agents Reshape Knowledge Work: Autonomy, Efficiency, and Scopecited by 1
  35. 2026Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Studycited by 1
  36. 2026SemPiper: Interactive Code Synthesis for Semantic Operators in Machine Learning Pipelinescited by 1
  37. 2026From Determinism to Delegation: AI-Native Software Engineering and the Evolution of the Agentic Engineercited by 1
  38. 2026Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLIcited by 1

50 most cited of 101 All 101 (JSON)