# Source Code Summarization with Structural Relative Position Guided Transformer

arXiv: https://arxiv.org/abs/2202.06521 · HTML: https://mlsanity.com/paper/2202.06521/

## Key facts

- Cited by 14 papers in the map (data through 7 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [A parallel corpus of Python functions and documentation strings for automated code documentation and code generation](https://mlsanity.com/paper/1707.02275/) (2017)
- [Automatically Generating Commit Messages from Diffs using Neural Machine Translation](https://mlsanity.com/paper/1708.09492/) (2017)
- [Multi-Modal Attention Network Learning for Semantic Source Code Retrieval](https://mlsanity.com/paper/1909.13516/) (2019)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [Incorporating External Knowledge through Pre-training for Natural Language to Code Generation](https://mlsanity.com/paper/2004.09015/) (2020)
- [Learning to Update Natural Language Comments Based on Code Changes](https://mlsanity.com/paper/2004.12169/) (2020)
- [Code Summarization with Structure-induced Transformer](https://mlsanity.com/paper/2012.14710/) (2020)
- [Tree-to-Sequence Attentional Neural Machine Translation](https://mlsanity.com/paper/1603.06075/) (2016)
- [Automatic Source Code Summarization with Extended Tree-LSTM](https://mlsanity.com/paper/1906.08094/) (2019)
- [A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes](https://mlsanity.com/paper/1704.04856/) (2017)
- [Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting](https://mlsanity.com/paper/2103.07845/) (2021)
- [Automatic Generation of Text Descriptive Comments for Code Blocks](https://mlsanity.com/paper/1808.06880/) (2018)
- [Leveraging Code Generation to Improve Code Retrieval and Summarization via Dual Learning](https://mlsanity.com/paper/2002.10198/) (2020)

## Built on by (most cited first, 14 of 14)

- [Automatic Code Summarization via ChatGPT: How Far Are We?](https://mlsanity.com/paper/2305.12865/) (2023)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [An Extractive-and-Abstractive Framework for Source Code Summarization](https://mlsanity.com/paper/2206.07245/) (2022)
- [Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers](https://mlsanity.com/paper/2302.01925/) (2023)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Tram: A Token-level Retrieval-augmented Mechanism for Source Code Summarization](https://mlsanity.com/paper/2305.11074/) (2023)
- [ESALE: Enhancing Code-Summary Alignment Learning for Source Code Summarization](https://mlsanity.com/paper/2407.01646/) (2024)
- [Abstract Syntax Tree for Programming Language Understanding and Representation: How Far Are We?](https://mlsanity.com/paper/2312.00413/) (2023)
- [The Role of Transformer Models in Advancing Blockchain Technology: A Systematic Survey](https://mlsanity.com/paper/2409.02139/) (2024)
- [PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference](https://mlsanity.com/paper/2409.15104/) (2024)
- [FDC: Fast KV Dimensionality Compression for Efficient LLM Inference](https://mlsanity.com/paper/2408.04107/) (2024)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [Revisiting the Role of Natural Language Code Comments in Code Translation](https://mlsanity.com/paper/2601.16661/) (2026)
- [Syntax vs. Semantics: How Transformers Learn Deep Dependencies](https://mlsanity.com/paper/2608.26139/) (2026)

## Abstract (arXiv)

Source code summarization aims at generating concise and clear natural language descriptions for programming languages. Well-written code summaries are beneficial for programmers to participate in the software development and maintenance process. To learn the semantic representations of source code, recent efforts focus on incorporating the syntax structure of code into neural networks such as Transformer. Such Transformer-based approaches can better capture the long-range dependencies than other neural networks including Recurrent Neural Networks (RNNs), however, most of them do not consider the structural relative correlations between tokens, e.g., relative positions in Abstract Syntax Trees (ASTs), which is beneficial for code semantics learning. To model the structural dependency, we propose a Structural Relative Position guided Transformer, named SCRIPT. SCRIPT first obtains the structural relative positions between tokens via parsing the ASTs of source code, and then passes them into two types of Transformer encoders. One Transformer directly adjusts the input according to the structural relative distance; and the other Transformer encodes the structural relative positions during computing the self-attention scores. Finally, we stack these two types of Transformer encoders to learn representations of source code. Experimental results show that the proposed SCRIPT outperforms the state-of-the-art methods by at least 1.6%, 1.4% and 2.8% with respect to BLEU, ROUGE-L and METEOR on benchmark datasets, respectively. We further show that how the proposed SCRIPT captures the structural relative dependencies.