mlsanityData to 14 Oct 2026

AST-T5: Structure-Aware Pretraining for Code Generation and Understanding

arXiv 2401.03003 ↗First posted 5 Jan 2024cs.SE cs.CL cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have made significant advancements in code-related tasks, yet many LLMs treat code as simple sequences, neglecting its structured nature. We introduce AST-T5, a novel pretraining paradigm that leverages the Abstract Syntax Tree (AST) for enhanced code generation, transpilation, and understanding. Using dynamic programming, our AST-Aware Segmentation retains code structure, while our AST-Aware Span Corruption objective equips the model to reconstruct various code structures. Unlike other models, AST-T5 avoids intricate program analyses or architectural changes, so it integrates seamlessly with any encoder-decoder Transformer. Evaluations show that AST-T5 consistently outperforms similar-sized LMs across various code-related tasks. Structure-awareness makes AST-T5 particularly powerful in code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in CodeXGLUE. Our code and model are publicly available at https://github.com/gonglinyuan/ast_t5.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

40 of 48 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
28 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  8. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  15. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  16. 2020Unsupervised Translation of Programming Languagescited by 263
  17. 2017Learning to Represent Programs with Graphscited by 220
  18. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  19. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  20. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 108
  21. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  22. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  23. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  24. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  25. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  26. 2017Code Completion with Neural Attention and Pointer Networkscited by 49
  27. 2021Latent Execution for Neural Program Synthesiscited by 35
  28. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 15; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 61
  3. 2024RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generationcited by 26
  4. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
  5. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
  6. 2025cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Treecited by 17
  7. 2024ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generationcited by 14
  8. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  9. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generationcited by 10
  10. 2024LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issuescited by 10
  11. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  12. 2025NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debuggingcited by 4
21 more
  1. 2025Structure-Aware Fill-in-the-Middle Pretraining for Codecited by 4
  2. 2025Alignment with Fill-In-the-Middle for Enhancing Code Generationcited by 4
  3. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  4. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  5. 2026Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requestscited by 3
  6. 2025JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generationcited by 2
  7. 2025MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schemacited by 2
  8. 2026Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Modelscited by 2
  9. 2024Insights from Benchmarking Frontier Language Models on Web App Code Generationcited by 1
  10. 2024ASGDiffusion: Parallel High-Resolution Generation with Asynchronous Structure Guidancecited by 1
  11. 2025Detecting Stealthy Data Poisoning Attacks in AI Code Generatorscited by 1
  12. 2025Vector Graph-Based Repository Understanding for Issue-Driven File Retrievalcited by 1
  13. 2025Understanding Network Behaviors through Natural Language Question-Answeringcited by 1
  14. 2026Q-Bridge: Code Translation for Quantum Machine Learning via LLMscited by 1
  15. 2025Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completioncited by 0
  16. 2025app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffoldingcited by 0
  17. 2025REMODEL-LLM: Transforming C code to Java using LLMscited by 0
  18. 2026Revisiting Gene Ontology Knowledge Discovery with Hierarchical Feature Selection and Virtual Study Group of AI Agentscited by 0
  19. 2026Static Program Slicing Using Language Models With Dataflow-Aware Pretraining and Constrained Decodingcited by 0
  20. 2026Dependency-Guided Code Generation: Structured Matrix Decomposition and Consistency-Guided Refinementcited by 0
  21. 2026SGA: Plug&Play Geometric Verification for Educational Video Synthesiscited by 0