What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents are black boxes: we cannot inspect how they generate code, but we can inspect what they change. This distinction matters for search-based software engineering (SBSE), where techniques such as genetic improvement (in the performance-optimisation application we study) depend on mutation operators that reflect how code is actually transformed. Fewer than 1% of the 33,596 agent PRs in AIDev-pop target performance, making each case a rare window into otherwise opaque agent behaviour. We classify 1,254 performance-relevant diff hunks from 216 of these PRs, spanning five agent systems, against the 18-category syntactic mutation taxonomy of Even-Mendoza et al. (2025) using a dual-LLM intersection pipeline. Three categories dominate: name modification (37.0%), object creation (26.4%), and type change (22.7%), a profile markedly different from prior GI corpora where no change accounted for 84%. Each agent's deployed system commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint category subset. Agent identity and target strategy are therefore informative priors that narrow the effective SBSE operator space. Replication package: https://github.com/5uper6rain/ssbse-challenge-2026
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 4 of its 8 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
4 of 8 references.
Closest ancestors · 1
Same direction, newest first
Other references · 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Self-Evolving Coding Agents
- 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
- 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
- 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
- 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests
- 2026What Do Evolutionary Coding Agents Evolve?
- 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
- 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time