# A Transformer-based Approach for Source Code Summarization

arXiv: https://arxiv.org/abs/2005.00653 · HTML: https://mlsanity.com/paper/2005.00653/

## Key facts

- Cited by 133 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 30 of 133)

- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [AVATAR: A Parallel Corpus for Java-Python Program Translation](https://mlsanity.com/paper/2108.11590/) (2021)
- [NatGen: Generative pre-training by "Naturalizing" source code](https://mlsanity.com/paper/2206.07585/) (2022)
- [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021)
- [CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning](https://mlsanity.com/paper/2110.12925/) (2021)
- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [Can Large Language Models Write Parallel Code?](https://mlsanity.com/paper/2401.12554/) (2024)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023)
- [Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review](https://mlsanity.com/paper/2307.02503/) (2023)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [Distilled GPT for Source Code Summarization](https://mlsanity.com/paper/2308.14731/) (2023)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [CodeQA: A Question Answering Dataset for Source Code Comprehension](https://mlsanity.com/paper/2109.08365/) (2021)
- [Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains](https://mlsanity.com/paper/2402.05140/) (2024)
- [Symbolic Behaviour in Artificial Intelligence](https://mlsanity.com/paper/2102.03406/) (2021)
- [On Multi-Modal Learning of Editing Source Code](https://mlsanity.com/paper/2108.06645/) (2021)
- [Counterfactual Explanations for Models of Code](https://mlsanity.com/paper/2111.05711/) (2021)
- [Project-Level Encoding for Neural Source Code Summarization of Subroutines](https://mlsanity.com/paper/2103.11599/) (2021)

## Abstract (arXiv)

Generating a readable summary that describes the functionality of a program is known as source code summarization. In this task, learning code representation by modeling the pairwise relationship between code tokens to capture their long-range dependencies is crucial. To learn code representation for summarization, we explore the Transformer model that uses a self-attention mechanism and has shown to be effective in capturing long-range dependencies. In this work, we show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin. We perform extensive analysis and ablation studies that reveal several important findings, e.g., the absolute encoding of source code tokens' position hinders, while relative encoding significantly improves the summarization performance. We have made our code publicly available to facilitate future research.