# StructCoder: Structure-Aware Transformer for Code Generation

arXiv: https://arxiv.org/abs/2206.05239 · HTML: https://mlsanity.com/paper/2206.05239/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [DOBF: A Deobfuscation Pre-Training Objective for Programming Languages](https://mlsanity.com/paper/2102.07492/) (2021)
- [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://mlsanity.com/paper/2103.11318/) (2021)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [Generative Code Modeling with Graphs](https://mlsanity.com/paper/1805.08490/) (2018)
- [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019)
- [Code Completion with Neural Attention and Pointer Networks](https://mlsanity.com/paper/1711.09573/) (2017)
- [Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning](https://mlsanity.com/paper/2105.14796/) (2021)
- [Exploring Dynamic Selection of Branch Expansion Orders for Code Generation](https://mlsanity.com/paper/2106.00261/) (2021)

## Built on by (most cited first, 30 of 34)

- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [AST-T5: Structure-Aware Pretraining for Code Generation and Understanding](https://mlsanity.com/paper/2401.03003/) (2024)
- [AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation](https://mlsanity.com/paper/2410.24117/) (2024)
- [Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks](https://mlsanity.com/paper/2505.16901/) (2025)
- [IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators](https://mlsanity.com/paper/2403.03894/) (2024)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis](https://mlsanity.com/paper/2412.14234/) (2024)
- [Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models](https://mlsanity.com/paper/2407.00456/) (2024)
- [CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning](https://mlsanity.com/paper/2504.14119/) (2025)
- [Understanding Programs by Exploiting (Fuzzing) Test Cases](https://mlsanity.com/paper/2305.13592/) (2023)
- [Program Translation via Code Distillation](https://mlsanity.com/paper/2310.11476/) (2023)
- [RAMBO: Enhancing RAG-based Repository-Level Method Body Completion](https://mlsanity.com/paper/2409.15204/) (2024)
- [Multilingual Code Co-Evolution Using Large Language Models](https://mlsanity.com/paper/2307.14991/) (2023)
- [Code Representation Pre-training with Complements from Program Executions](https://mlsanity.com/paper/2309.09980/) (2023)
- [Exploring Data Augmentation for Code Generation Tasks](https://mlsanity.com/paper/2302.03499/) (2023)
- [GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding](https://mlsanity.com/paper/2409.04183/) (2024)
- [Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis](https://mlsanity.com/paper/2310.13669/) (2023)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models](https://mlsanity.com/paper/2401.10716/) (2024)
- [Enhancing AI-based Generation of Software Exploits with Contextual Information](https://mlsanity.com/paper/2408.02402/) (2024)
- [Learning Metadata-Agnostic Representations for Text-to-SQL In-Context Example Selection](https://mlsanity.com/paper/2410.14049/) (2024)
- [Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation](https://mlsanity.com/paper/2501.16050/) (2025)
- [SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches](https://mlsanity.com/paper/2508.01237/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text](https://mlsanity.com/paper/2306.06371/) (2023)
- [ASTormer: An AST Structure-aware Transformer Decoder for Text-to-SQL](https://mlsanity.com/paper/2310.18662/) (2023)
- [ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositories](https://mlsanity.com/paper/2604.07341/) (2026)
- [An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code](https://mlsanity.com/paper/2307.02443/) (2023)
- [MCGMark: An Encodable and Robust Online Watermark for Tracing LLM-Generated Malicious Code](https://mlsanity.com/paper/2408.01354/) (2024)

## Abstract (arXiv)

There has been a recent surge of interest in automating software engineering tasks using deep learning. This paper addresses the problem of code generation, where the goal is to generate target code given source code in a different language or a natural language description. Most state-of-the-art deep learning models for code generation use training strategies primarily designed for natural language. However, understanding and generating code requires a more rigorous comprehension of the code syntax and semantics. With this motivation, we develop an encoder-decoder Transformer model where both the encoder and decoder are explicitly trained to recognize the syntax and data flow in the source and target codes, respectively. We not only make the encoder structure-aware by leveraging the source code's syntax tree and data flow graph, but we also support the decoder in preserving the syntax and data flow of the target code by introducing two novel auxiliary tasks: AST (Abstract Syntax Tree) paths prediction and data flow prediction. To the best of our knowledge, this is the first work to introduce a structure-aware Transformer decoder that models both syntax and data flow to enhance the quality of generated code. The proposed StructCoder model achieves state-of-the-art performance on code translation and text-to-code generation tasks in the CodeXGLUE benchmark, and improves over baselines of similar size on the APPS code generation benchmark. Our code is publicly available at https://github.com/reddy-lab-code-research/StructCoder/.