# Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests

arXiv: https://arxiv.org/abs/2601.17627 · HTML: https://mlsanity.com/paper/2601.17627/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 24 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 18 of 18)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4](https://mlsanity.com/paper/2403.02839/) (2024)
- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://mlsanity.com/paper/2402.16829/) (2024)
- [AST-T5: Structure-Aware Pretraining for Code Generation and Understanding](https://mlsanity.com/paper/2401.03003/) (2024)
- [RACE: Retrieval-Augmented Commit Message Generation](https://mlsanity.com/paper/2203.02700/) (2022)
- [What Makes a Good Commit Message?](https://mlsanity.com/paper/2202.02974/) (2022)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity](https://mlsanity.com/paper/2508.21634/) (2025)
- [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024)
- [An Empirical Study on Commit Message Generation using LLMs via In-Context Learning](https://mlsanity.com/paper/2502.18904/) (2025)
- [AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans](https://mlsanity.com/paper/2509.21891/) (2025)
- [Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metrics](https://mlsanity.com/paper/2508.08661/) (2025)
- [Checklist Engineering Empowers Multilingual LLM Judges](https://mlsanity.com/paper/2507.06774/) (2025)
- [Contextual Code Retrieval for Commit Message Generation: A Preliminary Study](https://mlsanity.com/paper/2507.17690/) (2025)
- [Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach](https://mlsanity.com/paper/2408.00921/) (2024)
- [Evaluating the Impact of Data Cleaning on the Quality of Generated Pull Request Descriptions](https://mlsanity.com/paper/2505.01120/) (2025)

## Built on by (most cited first, 3 of 3)

- [AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?](https://mlsanity.com/paper/2607.03332/) (2026)
- [3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse](https://mlsanity.com/paper/2607.07980/) (2026)
- [AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints](https://mlsanity.com/paper/2608.00966/) (2026)

## Abstract (arXiv)

AI coding agents can autonomously generate pull requests (PRs), yet little is known about how their contributions compare to those of humans. We analyze 33,596 agent-generated PRs (APRs) and 6,618 human PRs (HPRs) to compare code-change characteristics and message quality. We observe that APR-introduced symbols (functions and classes) are removed much sooner than those in HPRs (median time to removal 3 vs. 34 days) and are also removed more often (symbol churn 7.33% vs. 4.10%), reflecting a focus on other tasks like documentation and test updates. Agents generate stronger commit-level messages (semantic similarity 0.72 vs. 0.68) but lag humans at PR-level summarization (PR-commit similarity 0.86 vs. 0.88). Commit message length is the best predictor of description quality, indicating reliance on individual commits over full-PR reasoning. These findings highlight a gap between agents' micro-level precision and macro-level communication, suggesting opportunities to improve agent-driven development workflows.