# AST-Transformer: Encoding Abstract Syntax Trees Efficiently for Code Summarization

arXiv: https://arxiv.org/abs/2112.01184 · HTML: https://mlsanity.com/paper/2112.01184/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Improving Automatic Source Code Summarization via Deep Reinforcement Learning](https://mlsanity.com/paper/1811.07234/) (2018)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [Tree-to-Sequence Attentional Neural Machine Translation](https://mlsanity.com/paper/1603.06075/) (2016)

## Built on by (most cited first, 11 of 11)

- [xPQA: Cross-Lingual Product Question Answering across 12 Languages](https://mlsanity.com/paper/2305.09249/) (2023)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Automating Code-Related Tasks Through Transformers: The Impact of Pre-training](https://mlsanity.com/paper/2302.04048/) (2023)
- [A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks](https://mlsanity.com/paper/2312.15614/) (2023)
- [InternLM-Law: An Open Source Chinese Legal Large Language Model](https://mlsanity.com/paper/2406.14887/) (2024)
- [Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer](https://mlsanity.com/paper/2208.11523/) (2022)
- [MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation](https://mlsanity.com/paper/2403.17357/) (2024)
- [FuncEvalGMN: Evaluating Functional Correctness of SQL via Graph Matching Network](https://mlsanity.com/paper/2407.14530/) (2024)
- [Bringing Structure to Naturalness: On the Naturalness of ASTs](https://mlsanity.com/paper/2504.08234/) (2025)
- [Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation](https://mlsanity.com/paper/2506.14649/) (2025)
- [Learning Sparse Decision Trees via Transformer Variational Auto-Encoders](https://mlsanity.com/paper/2609.01430/) (2026)

## Abstract (arXiv)

Code summarization aims to generate brief natural language descriptions for source code. As source code is highly structured and follows strict programming language grammars, its Abstract Syntax Tree (AST) is often leveraged to inform the encoder about the structural information. However, ASTs are usually much longer than the source code. Current approaches ignore the size limit and simply feed the whole linearized AST into the encoder. To address this problem, we propose AST-Transformer to efficiently encode tree-structured ASTs. Experiments show that AST-Transformer outperforms the state-of-arts by a substantial margin while being able to reduce 90~95% of the computational complexity in the encoding process.