# Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity

arXiv: https://arxiv.org/abs/2601.18345 · HTML: https://mlsanity.com/paper/2601.18345/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 63 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 21 of 21)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://mlsanity.com/paper/2507.09089/) (2025)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Language Models for Code Completion: A Practical Evaluation](https://mlsanity.com/paper/2402.16197/) (2024)
- [Investigating and Designing for Trust in AI-powered Code Generation Tools](https://mlsanity.com/paper/2305.11248/) (2023)
- [Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review](https://mlsanity.com/paper/2510.00328/) (2025)
- [Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study](https://mlsanity.com/paper/2402.16480/) (2024)
- [A User-centered Security Evaluation of Copilot](https://mlsanity.com/paper/2308.06587/) (2023)
- [DevGPT: Studying Developer-ChatGPT Conversations](https://mlsanity.com/paper/2309.03914/) (2023)
- [Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild](https://mlsanity.com/paper/2506.12347/) (2025)

## Built on by (most cited first, 6 of 6)

- [How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions](https://mlsanity.com/paper/2605.29442/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [Agentic Repository Mining: A Multi-Task Evaluation](https://mlsanity.com/paper/2605.04845/) (2026)
- [Agentic Very Much! Adoption of Coding Agent in New GitHub Projects](https://mlsanity.com/paper/2606.07448/) (2026)
- [On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces](https://mlsanity.com/paper/2608.28497/) (2026)
- [Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance](https://mlsanity.com/paper/2609.05677/) (2026)

## Abstract (arXiv)

In 2025, coding agents have seen a very rapid adoption. Coding agents leverage Large Language Models (LLMs) in ways that are markedly different from LLM-based code completion, making their study critical. Moreover, unlike LLM-based completion, coding agents leave visible traces in software repositories, enabling the use of MSR techniques to study their impact on SE practices. This paper documents the promises, perils, and heuristics that we have gathered from studying coding agent activity on GitHub.