mlsanityData to 14 Oct 2026

Recommendations for Datasets for Source Code Summarization

arXiv 1904.02660 ↗First posted 4 Apr 2019cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Source Code Summarization is the task of writing short, natural language descriptions of source code. The main use for these descriptions is in software documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code summarization is rapidly becoming a popular research problem, but progress is restrained due to a lack of suitable datasets. In addition, a lack of community standards for creating datasets leads to confusing and unreproducible research results -- we observe swings in performance of more than 33% due only to changes in dataset design. In this paper, we make recommendations for these standards from experimental results. We release a dataset based on prior work of over 2.1m pairs of Java methods and one sentence method descriptions from over 28k Java projects. We describe the dataset and point out key differences from natural language data, to guide and support future researchers.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 11; 2023: 14; 2024: 5; 2025: 1; 2026: 3.

46 papers.

Sort:

By area:

  1. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  2. 2021On the Evaluation of Neural Code Summarizationcited by 48
  3. 2023Distilled GPT for Source Code Summarizationcited by 31
  4. 2023Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binariescited by 26
  5. 2021Project-Level Encoding for Neural Source Code Summarization of Subroutinescited by 24
  6. 2021A Survey on Machine Learning Techniques for Source Code Analysiscited by 24
  7. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  8. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  9. 2021On the Evaluation of Commit Message Generation Models: An Experimental Studycited by 20
  10. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  11. 2022An Extractive-and-Abstractive Framework for Source Code Summarizationcited by 18
  12. 2024Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarizationcited by 16
34 more
  1. 2022Pop Quiz! Can a Large Language Model Help With Reverse Engineering?cited by 14
  2. 2022M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarizationcited by 14
  3. 2022A Survey on Pretrained Language Models for Neural Code Intelligencecited by 13
  4. 2022Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languagescited by 12
  5. 2021Impact of Evaluation Methodologies on Code Summarizationcited by 11
  6. 2024EyeTrans: Merging Human and Machine Attention for Neural Code Summarizationcited by 9
  7. 2023Automating Code-Related Tasks Through Transformers: The Impact of Pre-trainingcited by 8
  8. 2023Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarizationcited by 8
  9. 2024ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarizationcited by 8
  10. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 7
  11. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  12. 2021HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebookscited by 6
  13. 2021Code to Comment Translation: A Comparative Study on Model Effectiveness & Errorscited by 6
  14. 2021A Neural Question Answering System for Basic Questions about Subroutinescited by 5
  15. 2021Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approachcited by 5
  16. 2021Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understandingcited by 5
  17. 2023Semantic Similarity Loss for Neural Source Code Summarizationcited by 5
  18. 2022A Survey of Deep Learning Models for Structural Code Understandingcited by 4
  19. 2023TPDR: A Novel Two-Step Transformer-based Product and Class Description Match and Retrieval Methodcited by 4
  20. 2024Do Current Language Models Support Code Intelligence for R Programming Language?cited by 4
  21. 2022GPT-based Open-Ended Knowledge Tracingcited by 3
  22. 2023Revisiting File Context for Source Code Summarizationcited by 3
  23. 2022JEMMA: An Extensible Java Dataset for ML4Code Applicationscited by 2
  24. 2023Statement-based Memory for Neural Source Code Summarizationcited by 2
  25. 2021Neural Code Summarizationcited by 1
  26. 2025CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurementcited by 1
  27. 2026Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentationcited by 1
  28. 2022Meta Learning for Code Summarizationcited by 0
  29. 2022Preprocessing Source Code Comments for Linguistic Modelscited by 0
  30. 2023A Machine Learning Approach Towards SKILL Code Autocompletioncited by 0
  31. 2023Predicting Line-Level Defects by Capturing Code Contexts with Hierarchical Transformerscited by 0
  32. 2024In-Context Code-Text Learning for Bimodal Software Engineeringcited by 0
  33. 2026On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Studycited by 0
  34. 2026Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Developmentcited by 0