# Recommendations for Datasets for Source Code Summarization

arXiv: https://arxiv.org/abs/1904.02660 · HTML: https://mlsanity.com/paper/1904.02660/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 30 of 46)

- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries](https://mlsanity.com/paper/2301.01701/) (2023)
- [Project-Level Encoding for Neural Source Code Summarization of Subroutines](https://mlsanity.com/paper/2103.11599/) (2021)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [CoDesc: A Large Code-Description Parallel Dataset](https://mlsanity.com/paper/2105.14220/) (2021)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [On the Evaluation of Commit Message Generation Models: An Experimental Study](https://mlsanity.com/paper/2107.05373/) (2021)
- [The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation](https://mlsanity.com/paper/2305.06156/) (2023)
- [An Extractive-and-Abstractive Framework for Source Code Summarization](https://mlsanity.com/paper/2206.07245/) (2022)
- [Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization](https://mlsanity.com/paper/2402.14182/) (2024)
- [Pop Quiz! Can a Large Language Model Help With Reverse Engineering?](https://mlsanity.com/paper/2202.01142/) (2022)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [A Survey on Pretrained Language Models for Neural Code Intelligence](https://mlsanity.com/paper/2212.10079/) (2022)
- [Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2205.11116/) (2022)
- [Impact of Evaluation Methodologies on Code Summarization](https://mlsanity.com/paper/2108.09619/) (2021)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Automating Code-Related Tasks Through Transformers: The Impact of Pre-training](https://mlsanity.com/paper/2302.04048/) (2023)
- [Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization](https://mlsanity.com/paper/2305.09773/) (2023)
- [ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization](https://mlsanity.com/paper/2407.01646/) (2024)
- [A Language Model of Java Methods with Train/Test Deduplication](https://mlsanity.com/paper/2305.08286/) (2023)
- [Assessing Evaluation Metrics for Neural Test Oracle Generation](https://mlsanity.com/paper/2310.07856/) (2023)
- [HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks](https://mlsanity.com/paper/2104.01002/) (2021)
- [Code to Comment Translation: A Comparative Study on Model Effectiveness & Errors](https://mlsanity.com/paper/2106.08415/) (2021)
- [A Neural Question Answering System for Basic Questions about Subroutines](https://mlsanity.com/paper/2101.03999/) (2021)
- [Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach](https://mlsanity.com/paper/2103.11448/) (2021)
- [Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understanding](https://mlsanity.com/paper/2109.07452/) (2021)
- [Semantic Similarity Loss for Neural Source Code Summarization](https://mlsanity.com/paper/2308.07429/) (2023)
- [A Survey of Deep Learning Models for Structural Code Understanding](https://mlsanity.com/paper/2205.01293/) (2022)

## Abstract (arXiv)

Source Code Summarization is the task of writing short, natural language descriptions of source code. The main use for these descriptions is in software documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code summarization is rapidly becoming a popular research problem, but progress is restrained due to a lack of suitable datasets. In addition, a lack of community standards for creating datasets leads to confusing and unreproducible research results -- we observe swings in performance of more than 33% due only to changes in dataset design. In this paper, we make recommendations for these standards from experimental results. We release a dataset based on prior work of over 2.1m pairs of Java methods and one sentence method descriptions from over 28k Java projects. We describe the dataset and point out key differences from natural language data, to guide and support future researchers.