mlsanityData to 7 Oct 2026

Distilled GPT for Source Code Summarization

arXiv 2308.14731 ↗First posted 28 Aug 2023cs.SE cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

A code summary is a brief natural language description of source code. Summaries are usually only a single sentence long, and yet form the backbone of developer documentation. A short descriptions such as "changes all visible polygons to the color blue" can give a programmer a high-level idea of what code does without the effort of reading the code itself. Recently, products based on Large Language Models such as ChatGPT have demonstrated a strong ability to write these descriptions automatically. However, to use these tools, programmers must send their code to untrusted third parties for processing (e.g., via an API call). This loss of custody is not acceptable to many organizations. In this paper, we present an alternative: we train an open source model using sample output generated by GPT-3.5 in a process related to knowledge distillation. Our model is small enough (350m parameters) to be run on a single 16gb GPU, yet we show in our evaluation that it is large enough to mimic GPT-3.5 on this task.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 45 of its 76 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

45 of 76 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2021Scaling Vision Transformerscited by 993
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  11. 2023The False Promise of Imitating Proprietary LLMscited by 254
  12. 2018code2vec: Learning Distributed Representations of Codecited by 249
33 more
  1. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  2. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  3. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
  4. 2017Learning to Represent Programs with Graphscited by 219
  5. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  6. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  7. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  10. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  11. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  12. 2019Recommendations for Datasets for Source Code Summarizationcited by 46
  13. 2017Automatically Generating Commit Messages from Diffs using Neural Machine Translationcited by 42
  14. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  15. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  16. 2020Improved Automatic Summarization of Subroutines via Attention to File Contextcited by 34
  17. 2023Automatic Code Summarization via ChatGPT: How Far Are We?cited by 34
  18. 2023Phoenix: Democratizing ChatGPT across Languagescited by 32
  19. 2021Project-Level Encoding for Neural Source Code Summarization of Subroutinescited by 24
  20. 2017A Neural Architecture for Generating Natural Language Descriptions from Source Code Changescited by 23
  21. 2022Are We Building on the Rock? On the Importance of Data Preprocessing for Code Summarizationcited by 22
  22. 2021How much pretraining data do language models need to learn syntax?cited by 21
  23. 2020A Multi-Perspective Architecture for Semantic Code Searchcited by 20
  24. 2018Automatic Generation of Text Descriptive Comments for Code Blockscited by 18
  25. 2022Semantic Similarity Metrics for Evaluating Source Code Summarizationcited by 14
  26. 2023Distilling ChatGPT for Explainable Automated Student Answer Assessmentcited by 14
  27. 2021Action Word Prediction for Neural Source Code Summarizationcited by 11
  28. 2023Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domaincited by 8
  29. 2014Autofolding for Source Code Summarizationcited by 6
  30. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 6
  31. 2021A Neural Question Answering System for Basic Questions about Subroutinescited by 5
  32. 2023Autonomy Is An Acquired Taste: Exploring Developer Preferences for GitHub Botscited by 4
  33. 2021Do you really code? Designing and Evaluating Screening Questions for Online Surveys with Programmerscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 6; 2026: 11.

30 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  3. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  4. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  5. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  6. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  7. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  8. 2024Calibration of Large Language Models on Code Summarizationcited by 12
  9. 2024ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarizationcited by 6
  10. 2024XMainframe: A Large Language Model for Mainframe Modernizationcited by 6
  11. 2023The Behavior of Large Language Models When Prompted to Generate Code Explanationscited by 3
  12. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
18 more
  1. 2025Commenting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarizationcited by 3
  2. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  3. 2024Eliminating Backdoors in Neural Code Models for Secure Code Understandingcited by 2
  4. 2025Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Modelscited by 2
  5. 2026Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Developmentcited by 2
  6. 2026Evaluating Large Language Models for Security Bug Report Predictioncited by 1
  7. 2026Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebasescited by 1
  8. 2023ECAMP: Entity-centered Context-aware Medical Vision Language Pre-trainingcited by 0
  9. 2025AI-Mediated Code Comment Improvementcited by 0
  10. 2025Simple and Effective Baselines for Code Summarisation Evaluationcited by 0
  11. 2025Assertion-Aware Test Code Summarization with Large Language Modelscited by 0
  12. 2026On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Studycited by 0
  13. 2026EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarizationcited by 0
  14. 2026Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Modelscited by 0
  15. 2026XSearch: Explainable Code Search via Concept-to-Code Alignmentcited by 0
  16. 2026Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessmentcited by 0
  17. 2026Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Taskcited by 0
  18. 2026CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernizationcited by 0