# Code Structure Guided Transformer for Source Code Summarization

arXiv: https://arxiv.org/abs/2104.09340 · HTML: https://mlsanity.com/paper/2104.09340/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 74 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020)
- [Guiding Neural Machine Translation with Retrieved Translation Pieces](https://mlsanity.com/paper/1804.02559/) (2018)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [A parallel corpus of Python functions and documentation strings for automated code documentation and code generation](https://mlsanity.com/paper/1707.02275/) (2017)
- [Structured Neural Summarization](https://mlsanity.com/paper/1811.01824/) (2018)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [Code Summarization with Structure-induced Transformer](https://mlsanity.com/paper/2012.14710/) (2020)

## Built on by (most cited first, 30 of 30)

- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [Search-Based LLMs for Code Optimization](https://mlsanity.com/paper/2408.12159/) (2024)
- [ProTo: Program-Guided Transformer for Program-Guided Tasks](https://mlsanity.com/paper/2110.00804/) (2021)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunities](https://mlsanity.com/paper/2402.18062/) (2024)
- [An Exploratory Study on Code Attention in BERT](https://mlsanity.com/paper/2204.10200/) (2022)
- [JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models](https://mlsanity.com/paper/2406.12902/) (2024)
- [RAMBO: Enhancing RAG-based Repository-Level Method Body Completion](https://mlsanity.com/paper/2409.15204/) (2024)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization](https://mlsanity.com/paper/2407.01646/) (2024)
- [An Empirical Study on Capability of Large Language Models in Understanding Code Semantics](https://mlsanity.com/paper/2407.03611/) (2024)
- [Can Developers Prompt? A Controlled Experiment for Code Documentation Generation](https://mlsanity.com/paper/2408.00686/) (2024)
- [XMainframe: A Large Language Model for Mainframe Modernization](https://mlsanity.com/paper/2408.04660/) (2024)
- [GypSum: Learning Hybrid Representations for Code Summarization](https://mlsanity.com/paper/2204.12916/) (2022)
- [Semantic Similarity Loss for Neural Source Code Summarization](https://mlsanity.com/paper/2308.07429/) (2023)
- [RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry](https://mlsanity.com/paper/2505.15179/) (2025)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)
- [Improving Retrieval-Augmented Code Comment Generation by Retrieving for Generation](https://mlsanity.com/paper/2408.03623/) (2024)
- [Automatic Comment Generation via Multi-Pass Deliberation](https://mlsanity.com/paper/2209.06634/) (2022)
- [LIVABLE: Exploring Long-Tailed Classification of Software Vulnerability Types](https://mlsanity.com/paper/2306.06935/) (2023)
- [CSA-Trans: Code Structure Aware Transformer for AST](https://mlsanity.com/paper/2404.05767/) (2024)
- [The Role of Transformer Models in Advancing Blockchain Technology: A Systematic Survey](https://mlsanity.com/paper/2409.02139/) (2024)
- [Statement-based Memory for Neural Source Code Summarization](https://mlsanity.com/paper/2307.11709/) (2023)
- [Beyond Dataset Watermarking: Model-Level Copyright Protection for Code Summarization Models](https://mlsanity.com/paper/2410.14102/) (2024)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework](https://mlsanity.com/paper/2510.17163/) (2025)
- [Understanding Long Programming Languages with Structure-Aware Sparse Attention](https://mlsanity.com/paper/2205.13730/) (2022)
- [LSem2Vec: A Simple yet Effective Two-Stage Approach for Source Code Embedding](https://mlsanity.com/paper/2409.14644/) (2024)
- [Revisiting the Role of Natural Language Code Comments in Code Translation](https://mlsanity.com/paper/2601.16661/) (2026)

## Abstract (arXiv)

Code summaries help developers comprehend programs and reduce their time to infer the program functionalities during software maintenance. Recent efforts resort to deep learning techniques such as sequence-to-sequence models for generating accurate code summaries, among which Transformer-based approaches have achieved promising performance. However, effectively integrating the code structure information into the Transformer is under-explored in this task domain. In this paper, we propose a novel approach named SG-Trans to incorporate code structural properties into Transformer. Specifically, we inject the local symbolic information (e.g., code tokens and statements) and global syntactic structure (e.g., data flow graph) into the self-attention module of Transformer as inductive bias. To further capture the hierarchical characteristics of code, the local information and global structure are designed to distribute in the attention heads of lower layers and high layers of Transformer. Extensive evaluation shows the superior performance of SG-Trans over the state-of-the-art approaches. Compared with the best-performing baseline, SG-Trans still improves 1.4% and 2.0% in terms of METEOR score, a metric widely used for measuring generation quality, respectively on two benchmark datasets.