# On the Evaluation of NLP-based Models for Software Engineering

arXiv: https://arxiv.org/abs/2203.17166 · HTML: https://mlsanity.com/paper/2203.17166/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 19 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 5 of 5)

- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020)
- [Automatic Generation of Pull Request Descriptions](https://mlsanity.com/paper/1909.06987/) (2019)
- [Generating Question Titles for Stack Overflow from Mined Code Snippets](https://mlsanity.com/paper/2005.10157/) (2020)
- [Generating Summaries for Methods of Event-Driven Programs: an Android Case Study](https://mlsanity.com/paper/1812.04530/) (2018)

## Built on by (most cited first, 5 of 5)

- [Language Models for Code Completion: A Practical Evaluation](https://mlsanity.com/paper/2402.16197/) (2024)
- [Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study](https://mlsanity.com/paper/2304.12269/) (2023)
- [Semantically-enhanced Topic Recommendation System for Software Projects](https://mlsanity.com/paper/2206.00085/) (2022)
- [Dealing with Data for RE: Mitigating Challenges while using NLP and Generative AI](https://mlsanity.com/paper/2402.16977/) (2024)
- [An Empirical Study on Data Leakage and Generalizability of Link Prediction Models for Issues and Commits](https://mlsanity.com/paper/2211.00381/) (2022)

## Abstract (arXiv)

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: "Are these models evaluated fairly and consistently in the SE community?". To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.