# CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection

arXiv: https://arxiv.org/abs/2511.19875 · HTML: https://mlsanity.com/paper/2511.19875/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Builds on (most cited first, 30 of 31)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [RACE: Retrieval-Augmented Commit Message Generation](https://mlsanity.com/paper/2203.02700/) (2022)
- [A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes](https://mlsanity.com/paper/1704.04856/) (2017)
- [Deep Just-In-Time Inconsistency Detection Between Comments and Source Code](https://mlsanity.com/paper/2010.01625/) (2020)
- [On the Evaluation of Commit Message Generation Models: An Experimental Study](https://mlsanity.com/paper/2107.05373/) (2021)
- [What Makes a Good Commit Message?](https://mlsanity.com/paper/2202.02974/) (2022)
- [A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot](https://mlsanity.com/paper/2307.14397/) (2023)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [A Fine-grained Data Set and Analysis of Tangling in Bug Fixing Commits](https://mlsanity.com/paper/2011.06244/) (2020)
- [Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges](https://mlsanity.com/paper/2504.20799/) (2025)
- [Untangling Fine-Grained Code Changes](https://mlsanity.com/paper/1502.06757/) (2015)
- [Using Large Language Models for Commit Message Generation: A Preliminary Study](https://mlsanity.com/paper/2401.05926/) (2024)
- [Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond](https://mlsanity.com/paper/2404.14824/) (2024)
- [CommitBench: A Benchmark for Commit Message Generation](https://mlsanity.com/paper/2403.05188/) (2024)
- [Code Comment Inconsistency Detection with BERT and Longformer](https://mlsanity.com/paper/2207.14444/) (2022)
- [An Empirical Study on Commit Message Generation using LLMs via In-Context Learning](https://mlsanity.com/paper/2502.18904/) (2025)
- [DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistencies](https://mlsanity.com/paper/2306.06347/) (2023)
- [METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries](https://mlsanity.com/paper/2502.02794/) (2025)
- [Contextual Code Retrieval for Commit Message Generation: A Preliminary Study](https://mlsanity.com/paper/2507.17690/) (2025)
- [A Full-fledged Commit Message Quality Checker Based on Machine Learning](https://mlsanity.com/paper/2309.04797/) (2023)
- [Optimization is Better than Generation: Optimizing Commit Message Leveraging Human-written Commit Message](https://mlsanity.com/paper/2501.09861/) (2025)
- [CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency](https://mlsanity.com/paper/2506.20558/) (2025)
- [Evaluating Generated Commit Messages with Large Language Models](https://mlsanity.com/paper/2507.10906/) (2025)
- [SECOMlint: A linter for Security Commit Messages](https://mlsanity.com/paper/2301.06959/) (2023)

## Built on by (most cited first, 1 of 1)

- [Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests](https://mlsanity.com/paper/2601.04886/) (2026)

## Abstract (arXiv)

Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.