mlsanityData to 7 Oct 2026

Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests

arXiv 2601.17627 ↗First posted 24 Jan 2026cs.SEcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

AI coding agents can autonomously generate pull requests (PRs), yet little is known about how their contributions compare to those of humans. We analyze 33,596 agent-generated PRs (APRs) and 6,618 human PRs (HPRs) to compare code-change characteristics and message quality. We observe that APR-introduced symbols (functions and classes) are removed much sooner than those in HPRs (median time to removal 3 vs. 34 days) and are also removed more often (symbol churn 7.33% vs. 4.10%), reflecting a focus on other tasks like documentation and test updates. Agents generate stronger commit-level messages (semantic similarity 0.72 vs. 0.68) but lag humans at PR-level summarization (PR-commit similarity 0.86 vs. 0.88). Commit message length is the best predictor of description quality, indicating reliance on individual commits over full-PR reasoning. These findings highlight a gap between agents' micro-level precision and macro-level communication, suggesting opportunities to improve agent-driven development workflows.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 31 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

18 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans26 Sep
  2. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub18 Sep

Other references · 16

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  4. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  5. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 32
  6. 2022RACE: Retrieval-Augmented Commit Message Generationcited by 27
  7. 2022What Makes a Good Commit Message?cited by 19
  8. 2023From Commit Message Generation to History-Aware Commit Message Completioncited by 16
  9. 2025Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexitycited by 16
  10. 2024CommitBench: A Benchmark for Commit Message Generationcited by 8
  11. 2025An Empirical Study on Commit Message Generation using LLMs via In-Context Learningcited by 5
  12. 2025Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metricscited by 4
4 more
  1. 2025Checklist Engineering Empowers Multilingual LLM Judgescited by 3
  2. 2025Contextual Code Retrieval for Commit Message Generation: A Preliminary Studycited by 3
  3. 2024Automatic Pull Request Description Generation Using LLMs: A T5 Model Approachcited by 2
  4. 2025Evaluating the Impact of Data Cleaning on the Quality of Generated Pull Request Descriptionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?cited by 2
  2. 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
  2. 2026Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests
  3. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
  4. 2026Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice
  5. 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
  6. 2026How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses
  7. 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents
  8. 2026Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests