mlsanityData to 7 Oct 2026

Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests

arXiv 2601.04886 ↗First posted 8 Jan 2026cs.SE cs.AIcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that "descriptions claim unimplemented changes" was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5 times longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 33 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

9 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub18 Sep

Other references · 8

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  3. 2025Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspectivecited by 7
  4. 2021Bots Don't Mind Waiting, Do They? Comparing the Interaction With Automatically and Manually Created Pull Requestscited by 5
  5. 2024muPRL: A Mutation Testing Pipeline for Deep Reinforcement Learning based on Real Faultscited by 5
  6. 2025The Future of Generative AI in Software Engineering: A Vision from Industry and Academia in the European GENIUS Projectcited by 4
  7. 2025Enhancing Trust in Language Model-Based Code Optimization through RLHF: A Research Designcited by 1
  8. 2025CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detectioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026An Empirical Study of Downstream Adaptation for Agent Skillscited by 0
  2. 2026AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprintscited by 0
  3. 2026Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requestscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests
  2. 2026Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
  3. 2026Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice
  4. 2026How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
  5. 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents
  6. 2026Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
  7. 2026Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
  8. 2026Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests