# AST-T5: Structure-Aware Pretraining for Code Generation and Understanding

arXiv: https://arxiv.org/abs/2401.03003 · HTML: https://mlsanity.com/paper/2401.03003/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 48 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)

## Built on by (most cited first, 30 of 33)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement](https://mlsanity.com/paper/2411.00622/) (2024)
- [RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation](https://mlsanity.com/paper/2409.09584/) (2024)
- [Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks](https://mlsanity.com/paper/2403.04814/) (2024)
- [IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators](https://mlsanity.com/paper/2403.03894/) (2024)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation](https://mlsanity.com/paper/2405.17057/) (2024)
- [TreeDiff: AST-Guided Code Generation with Diffusion LLMs](https://mlsanity.com/paper/2508.01473/) (2025)
- [Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation](https://mlsanity.com/paper/2409.15895/) (2024)
- [LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues](https://mlsanity.com/paper/2411.13941/) (2024)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debugging](https://mlsanity.com/paper/2505.15356/) (2025)
- [Structure-Aware Fill-in-the-Middle Pretraining for Code](https://mlsanity.com/paper/2506.00204/) (2025)
- [Alignment with Fill-In-the-Middle for Enhancing Code Generation](https://mlsanity.com/paper/2508.19532/) (2025)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)
- [Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation](https://mlsanity.com/paper/2503.22776/) (2025)
- [Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests](https://mlsanity.com/paper/2601.17627/) (2026)
- [JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation](https://mlsanity.com/paper/2505.14978/) (2025)
- [MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema](https://mlsanity.com/paper/2510.16357/) (2025)
- [Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models](https://mlsanity.com/paper/2607.12463/) (2026)
- [Insights from Benchmarking Frontier Language Models on Web App Code Generation](https://mlsanity.com/paper/2409.05177/) (2024)
- [ASGDiffusion: Parallel High-Resolution Generation with Asynchronous Structure Guidance](https://mlsanity.com/paper/2412.06163/) (2024)
- [Detecting Stealthy Data Poisoning Attacks in AI Code Generators](https://mlsanity.com/paper/2508.21636/) (2025)
- [Vector Graph-Based Repository Understanding for Issue-Driven File Retrieval](https://mlsanity.com/paper/2510.08876/) (2025)
- [Understanding Network Behaviors through Natural Language Question-Answering](https://mlsanity.com/paper/2510.21894/) (2025)
- [Q-Bridge: Code Translation for Quantum Machine Learning via LLMs](https://mlsanity.com/paper/2603.27836/) (2026)
- [Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion](https://mlsanity.com/paper/2505.13073/) (2025)
- [app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffolding](https://mlsanity.com/paper/2509.03310/) (2025)
- [REMODEL-LLM: Transforming C code to Java using LLMs](https://mlsanity.com/paper/2512.11402/) (2025)
- [Revisiting Gene Ontology Knowledge Discovery with Hierarchical Feature Selection and Virtual Study Group of AI Agents](https://mlsanity.com/paper/2603.20132/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have made significant advancements in code-related tasks, yet many LLMs treat code as simple sequences, neglecting its structured nature. We introduce AST-T5, a novel pretraining paradigm that leverages the Abstract Syntax Tree (AST) for enhanced code generation, transpilation, and understanding. Using dynamic programming, our AST-Aware Segmentation retains code structure, while our AST-Aware Span Corruption objective equips the model to reconstruct various code structures. Unlike other models, AST-T5 avoids intricate program analyses or architectural changes, so it integrates seamlessly with any encoder-decoder Transformer. Evaluations show that AST-T5 consistently outperforms similar-sized LMs across various code-related tasks. Structure-awareness makes AST-T5 particularly powerful in code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in CodeXGLUE. Our code and model are publicly available at https://github.com/gonglinyuan/ast_t5.