# Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?

arXiv: https://arxiv.org/abs/2501.12617 · HTML: https://mlsanity.com/paper/2501.12617/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 85 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [On the "Naturalness" of Buggy Code](https://mlsanity.com/paper/1506.01159/) (2015)
- [Learning Natural Coding Conventions](https://mlsanity.com/paper/1402.4182/) (2014)
- [Towards Learning (Dis)-Similarity of Source Code from Program Contrasts](https://mlsanity.com/paper/2110.03868/) (2021)
- [Can Large Language Models Find And Fix Vulnerable Software?](https://mlsanity.com/paper/2308.10345/) (2023)
- [Commit2Vec: Learning Distributed Representations of Code Changes](https://mlsanity.com/paper/1911.07605/) (2019)
- [VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning](https://mlsanity.com/paper/2112.02650/) (2021)
- [Learning to Recommend Method Names with Global Context](https://mlsanity.com/paper/2201.10705/) (2022)
- [On the Generation, Structure, and Semantics of Grammar Patterns in Source Code Identifiers](https://mlsanity.com/paper/2007.08033/) (2020)
- [A Context-based Automated Approach for Method Name Consistency Checking and Suggestion](https://mlsanity.com/paper/2103.00269/) (2021)
- [On the Naming of Methods: A Survey of Professional Developers](https://mlsanity.com/paper/2102.13555/) (2021)
- [How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspective](https://mlsanity.com/paper/2308.12701/) (2023)

## Built on by (most cited first, 2 of 2)

- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [Multi-Agent Coordinated Rename Refactoring](https://mlsanity.com/paper/2601.00482/) (2026)

## Abstract (arXiv)

Concise and meaningful method names are crucial for program comprehension and maintenance. However, method names may become inconsistent with their corresponding implementations, causing confusion and errors. Several deep learning (DL)-based approaches have been proposed to identify such inconsistencies, with initial evaluations showing promising results. However, these evaluations typically use a balanced dataset, where the number of inconsistent and consistent names are equal. This setup, along with flawed dataset construction, leads to false positives, making reported performance less reliable in real-world scenarios, where most method names are consistent. In this paper, we present an empirical study that evaluates state-of-the-art DL-based methods for identifying inconsistent method names. We create a new benchmark by combining automatic identification from commit histories and manual developer inspections, reducing false positives. We evaluate five representative DL approaches (one retrieval-based and four generation-based) on this benchmark. Our results show that performance drops substantially when moving from the balanced dataset to the new benchmark. We further conduct quantitative and qualitative analyses to understand the strengths and weaknesses of the approaches. Retrieval-based methods perform well on simple methods and those with popular name sub-tokens but fail due to inefficient representation techniques. Generation-based methods struggle with inaccurate similarity calculations and immature name generation. Based on these findings, we propose improvements using contrastive learning and large language models (LLMs). Our study suggests that significant improvements are needed before these DL approaches can be effectively applied to real-world software systems.