# Are Coding Agents Generating Over-Mocked Tests? An Empirical Study arXiv: https://arxiv.org/abs/2602.00409 ยท HTML: https://mlsanity.com/paper/2602.00409/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 25 of its 58 references that are in the map. - First posted on arXiv 30 Jan 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 25 of 25) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024) - [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024) - [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023) - [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://mlsanity.com/paper/2507.09089/) (2025) - [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023) - [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025) - [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024) - [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024) - [Test-Driven Development for Code Generation](https://mlsanity.com/paper/2402.13521/) (2024) - [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024) - [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026) - [Refactoring Programs Using Large Language Models with Few-Shot Examples](https://mlsanity.com/paper/2311.11690/) (2023) - [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025) - [Sampling Projects in GitHub for MSR Studies](https://mlsanity.com/paper/2103.04682/) (2021) - [Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study](https://mlsanity.com/paper/2402.16480/) (2024) - [Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild](https://mlsanity.com/paper/2506.12347/) (2025) - [Untangling Fine-Grained Code Changes](https://mlsanity.com/paper/1502.06757/) (2015) - [Automatic Library Migration Using Large Language Models: First Results](https://mlsanity.com/paper/2408.16151/) (2024) - [Can GPT-4 Replicate Empirical Software Engineering Research?](https://mlsanity.com/paper/2310.01727/) (2023) - [Feature-Driven End-To-End Test Generation](https://mlsanity.com/paper/2408.01894/) (2024) - [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026) - [How Developers Engineer Test Cases: An Observational Study](https://mlsanity.com/paper/2103.01783/) (2021) - [Self-Admitted GenAI Usage in Open-Source Software](https://mlsanity.com/paper/2507.10422/) (2025) - [Automatically Removing Unnecessary Stubbings from Test Suites](https://mlsanity.com/paper/2407.20924/) (2024) ## Built on by (most cited first, 6 of 6) - [Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study](https://mlsanity.com/paper/2606.12231/) (2026) - [Agentic Repository Mining: A Multi-Task Evaluation](https://mlsanity.com/paper/2605.04845/) (2026) - [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026) - [Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?](https://mlsanity.com/paper/2607.28871/) (2026) - [Introducing Consort: A Spec-First Agent Framework for Enforced, Test-Driven Development on Live Database Branches](https://mlsanity.com/paper/2609.09671/) (2026) - [Chronicle: Cut-Point Replay for Regression Testing of LLM Agents](https://mlsanity.com/paper/2609.20625/) (2026) ## Abstract (arXiv) Coding agents have received significant adoption in software development recently. Unlike traditional LLM-based code completion tools, coding agents work with autonomy (e.g., invoking external tools) and leave visible traces in software repositories, such as authoring commits. Among their tasks, coding agents may autonomously generate software tests; however, the quality of these tests remains uncertain. In particular, excessive use of mocking can make tests harder to understand and maintain. This paper presents the first study to investigate the presence of mocks in agent-generated tests of real-world software systems. We analyzed over 1.2 million commits made in 2025 in 2,168 TypeScript, JavaScript, and Python repositories, including 48,563 commits by coding agents, 169,361 commits that modify tests, and 44,900 commits that add mocks to tests. Overall, we find that coding agents are more likely to modify tests and to add mocks to tests than non-coding agents. We detect that (1) 60% of the repositories with agent activity also contain agent test activity; (2) 23% of commits made by coding agents add/change test files, compared with 13% by non-agents; (3) 68% of the repositories with agent test activity also contain agent mock activity; (4) 36% of commits made by coding agents add mocks to tests, compared with 26% by non-agents; and (5) repositories created recently contain a higher proportion of test and mock commits made by agents. Finally, we conclude by discussing implications for developers and researchers. We call attention to the fact that tests with mocks may be potentially easier to generate automatically (but less effective at validating real interactions), and the need to include guidance on mocking practices in agent configuration files.