AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Coding agents often finish a session by returning one squashed patch that mixes feature implementation, bug fixes, refactorings, tests, and configuration edits. While the final code may be correct, collapsing unrelated edits into one patch removes the history structure needed for review, selective revert, and later maintenance. We study retrospective commit-history reconstruction: given a completed squashed change, an agent groups its hunks into commits and materializes a replayable commit sequence. We formalize the task as hunk-to-commit partitioning with a replay requirement and build AtomicCommitBench, containing 800 real consecutive-commit episodes from 10 Python projects. Because multiple decompositions may be reasonable, we evaluate outputs using complementary metrics: PPAR for replay validity, ARI for reference-based grouping quality, and TCR for failure containment on scoreable modified-test episodes. Natural retrospective reconstruction proves substantially harder than replay checking or synthetic tangling. Although nearly all non-random methods achieve replay validity (PPAR >= 0.988), grouping quality ranges from 0.03 to 0.46 ARI. Matched synthetic composites are much easier than real same-author squashed diffs (+0.333 ARI). In our evaluation, the GPT-5.4 setup driven by Codex CLI (0.46 ARI) and the GLM-5 setup driven by Claude Code (0.43 ARI) outperform MiniMax (0.31) and Kimi (0.29). Qualitative analysis identifies same-file lumping and support-hunk drift as recurring failure modes. Dependency-Aware Commit Evidence (DACE) improves the lower-scoring setups by 0.05 to 0.08 ARI, indicating that dependency cues and hunk-role information help agents avoid locality-driven grouping errors. AtomicCommitBench enables evaluation of the commit histories produced by coding agents alongside the final code.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 35 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
20 of 35 references.
Closest ancestors · 9
Same direction, newest first
- 2026Lore: Repurposing Git Commit Messages as a Structured Knowledge Protocol for AI Coding Agents16 Mar
- 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests24 Jan
- 2026Atomizer: An LLM-based Collaborative Multi-Agent Framework for Intent-Driven Commit Untangling3 Jan
- 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios20 Dec
- 2025HAFixAgent: History-Aware Program Repair Agent2 Nov
- 2025AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans26 Sep
- 2025GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git28 May
- 2025Code Researcher: Deep Research Agent for Large Systems Code and Commit History27 May
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 11
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2024AutoCodeRover: Autonomous Program Improvementcited by 272
- 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
- 2023From Commit Message Generation to History-Aware Commit Message Completioncited by 16
- 2020A Fine-grained Data Set and Analysis of Tangling in Bug Fixing Commitscited by 11
- 2024CommitBench: A Benchmark for Commit Message Generationcited by 8
- 2024RAG-Enhanced Commit Message Generationcited by 6
- 2025LLM-Driven Collaborative Model for Untangling Commits via Explicit and Implicit Dependency Reasoningcited by 2
- 2018The effects of change decomposition on code review -- a controlled experimentcited by 1
- 2026Detecting Multiple Semantic Concerns in Tangled Code Commitscited by 1
- 2026EditFlow: Benchmarking and Optimizing Code Edit Recommendation Systems via Reconstruction of Developer Flowscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026REVISE: Validity-Guided Recovery for Online Revisions in Agent Workflows
- 2026Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
- 2026VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
- 2026Learning to Commit: Generating Organic Pull Requests via Online Repository Memory
- 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue
- 2026Do These Violent Delights Have Violent Ends? Measuring the Post-Merge Fate of Agentic Code
- 2026SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
- 2026AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits