# GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git

arXiv: https://arxiv.org/abs/2505.22583 · HTML: https://mlsanity.com/paper/2505.22583/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [Sampling Projects in GitHub for MSR Studies](https://mlsanity.com/paper/2103.04682/) (2021)
- [Generating Commit Messages from Git Diffs](https://mlsanity.com/paper/1911.11690/) (2019)
- [Program Merge Conflict Resolution via Neural Transformers](https://mlsanity.com/paper/2109.00084/) (2021)
- [Can Program Synthesis be Used to Learn Merge Conflict Resolutions? An Empirical Analysis](https://mlsanity.com/paper/2103.02004/) (2021)

## Built on by (most cited first, 5 of 5)

- [SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments](https://mlsanity.com/paper/2507.09063/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?](https://mlsanity.com/paper/2607.03332/) (2026)
- [Merge-Bench: Resolve Merge Conflicts with Large Language Models](https://mlsanity.com/paper/2605.25890/) (2026)
- [DTOC: Dynamic Tool Output Compression for Adaptive Context Management in AI Agents](https://mlsanity.com/paper/2609.26121/) (2026)

## Abstract (arXiv)

Benchmarks for Software Engineering (SE) AI agents, most notably SWE-bench, have catalyzed progress in programming capabilities of AI agents. However, they overlook critical developer workflows such as Version Control System (VCS) operations. To address this issue, we present GitGoodBench, a novel benchmark for evaluating AI agent performance on VCS tasks. GitGoodBench covers three core Git scenarios extracted from permissive open-source Python, Java, and Kotlin repositories. Our benchmark provides three datasets: a comprehensive evaluation suite (900 samples), a rapid prototyping version (120 samples), and a training corpus (17,469 samples). We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall. We expect GitGoodBench to serve as a crucial stepping stone toward truly comprehensive SE agents that go beyond mere programming.