mlsanityData to 7 Oct 2026

Automatic Code Summarization via ChatGPT: How Far Are We?

arXiv 2305.12865 ↗First posted 22 May 2023cs.SE cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

To support software developers in understanding and maintaining programs, various automatic code summarization techniques have been proposed to generate a concise natural language comment for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of natural language processing tasks. Among them, ChatGPT is the most popular one which has attracted wide attention from the software engineering community. However, it still remains unclear how ChatGPT performs in (automatic) code summarization. Therefore, in this paper, we focus on evaluating ChatGPT on a widely-used Python dataset called CSN-Python and comparing it with several state-of-the-art (SOTA) code summarization models. Specifically, we first explore an appropriate prompt to guide ChatGPT to generate in-distribution comments. Then, we use such a prompt to ask ChatGPT to generate comments for all code snippets in the CSN-Python test set. We adopt three widely-used metrics (including BLEU, METEOR, and ROUGE-L) to measure the quality of the comments generated by ChatGPT and SOTA models (including NCS, CodeBERT, and CodeT5). The experimental results show that in terms of BLEU and ROUGE-L, ChatGPT's code summarization performance is significantly worse than all three SOTA models. We also present some cases and discuss the advantages and disadvantages of ChatGPT in code summarization. Based on the findings, we outline several open challenges and opportunities in ChatGPT-based code summarization.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 35 of its 68 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

35 of 68 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
23 more
  1. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  2. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  3. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  4. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  5. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  6. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 108
  7. 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
  8. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  9. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  10. 2021On the Evaluation of Neural Code Summarizationcited by 47
  11. 2019Retrieve and Refine: Exemplar-based Neural Comment Generationcited by 36
  12. 2020Improved Automatic Summarization of Subroutines via Attention to File Contextcited by 34
  13. 2020Code Summarization with Structure-induced Transformercited by 29
  14. 2019Automatic Source Code Summarization with Extended Tree-LSTMcited by 28
  15. 2020Code to Comment "Translation": Data, Metrics, Baselining & Evaluationcited by 28
  16. 2023A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repaircited by 26
  17. 2021Project-Level Encoding for Neural Source Code Summarization of Subroutinescited by 24
  18. 2021CoCoSum: Contextual Code Summarization with Multi-Relational Graph Neural Networkcited by 22
  19. 2021Improving Code Summarization with Block-wise Abstract Syntax Tree Splittingcited by 19
  20. 2021A Multi-Modal Transformer-based Code Summarization Approach for Smart Contractscited by 16
  21. 2022An Extractive-and-Abstractive Framework for Source Code Summarizationcited by 15
  22. 2022Source Code Summarization with Structural Relative Position Guided Transformercited by 14
  23. 2021API2Com: On the Improvement of Automatically Generated Code Comments Using API Documentationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 14; 2025: 8; 2026: 4.

34 papers.

Sort:

By area:

  1. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  2. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  3. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  4. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  5. 2023Distilled GPT for Source Code Summarizationcited by 30
  6. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  7. 2023The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Modelscited by 18
  8. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  9. 2023The potential of LLMs for coding with low-resource and domain-specific programming languagescited by 11
  10. 2024Compositional API Recommendation for Library-Oriented Code Generationcited by 10
  11. 2023A Prompt Learning Framework for Source Code Summarizationcited by 9
  12. 2023Just-in-Time Detection of Silent Security Patchescited by 7
22 more
  1. 2024Can Developers Prompt? A Controlled Experiment for Code Documentation Generationcited by 7
  2. 2024Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugscited by 7
  3. 2024ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarizationcited by 6
  4. 2023Semantic Similarity Loss for Neural Source Code Summarizationcited by 5
  5. 2024Curriculum Demonstration Selection for In-Context Learningcited by 5
  6. 2024Testing the Effect of Code Documentation on Large Language Model Code Understandingcited by 4
  7. 2024CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistantscited by 3
  8. 2025Commenting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarizationcited by 3
  9. 2026ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generationcited by 3
  10. 2024MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generationcited by 2
  11. 2025Zero-Shot Cross-Domain Code Search without Fine-Tuningcited by 2
  12. 2025Automatically Generating Rules of Malicious Software Packages via Large Language Modelcited by 2
  13. 2025Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Modelscited by 1
  14. 2025SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysiscited by 1
  15. 2026Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebasescited by 1
  16. 2024Unlocking Insights: Semantic Search in Jupyter Notebookscited by 0
  17. 2024Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generationcited by 0
  18. 2025Issue Retrieval and Verification Enhanced Supplementary Code Comment Generationcited by 0
  19. 2025ReDef: Do Code Language Models Truly Understand Code Changes for Just-in-Time Software Defect Prediction?cited by 0
  20. 2025Assertion-Aware Test Code Summarization with Large Language Modelscited by 0
  21. 2026Prompt-Driven Code Summarization: A Systematic Literature Reviewcited by 0
  22. 2026Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfercited by 0