# TreeGen: A Tree-Based Transformer Architecture for Code Generation

arXiv: https://arxiv.org/abs/1911.09983 · HTML: https://mlsanity.com/paper/1911.09983/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2019.

## Built on by (most cited first, 30 of 58)

- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation](https://mlsanity.com/paper/2501.08200/) (2025)
- [A Syntax-Guided Edit Decoder for Neural Program Repair](https://mlsanity.com/paper/2106.08253/) (2021)
- [CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning](https://mlsanity.com/paper/2110.12925/) (2021)
- [Large Language Model-Aware In-Context Learning for Code Generation](https://mlsanity.com/paper/2310.09748/) (2023)
- [TreeBERT: A Tree-Based Pre-Trained Model for Programming Language](https://mlsanity.com/paper/2105.12485/) (2021)
- [Automatic Programming: Large Language Models and Beyond](https://mlsanity.com/paper/2405.02213/) (2024)
- [StructCoder: Structure-Aware Transformer for Code Generation](https://mlsanity.com/paper/2206.05239/) (2022)
- [Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review](https://mlsanity.com/paper/2307.02503/) (2023)
- [Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models](https://mlsanity.com/paper/2308.10462/) (2023)
- [Transformers over Directed Acyclic Graphs](https://mlsanity.com/paper/2210.13148/) (2022)
- [IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators](https://mlsanity.com/paper/2403.03894/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs](https://mlsanity.com/paper/2301.12950/) (2023)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities](https://mlsanity.com/paper/2501.13742/) (2025)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair](https://mlsanity.com/paper/2302.01857/) (2023)
- [CodeEditor: Learning to Edit Source Code with Pre-trained Models](https://mlsanity.com/paper/2210.17040/) (2022)
- [Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation](https://mlsanity.com/paper/2208.10091/) (2022)
- [Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning](https://mlsanity.com/paper/2105.14796/) (2021)
- [Exploring Dynamic Selection of Branch Expansion Orders for Code Generation](https://mlsanity.com/paper/2106.00261/) (2021)
- [Exploiting Code Symmetries for Learning Program Semantics](https://mlsanity.com/paper/2308.03312/) (2023)
- [On the Effectiveness of Pretrained Models for API Learning](https://mlsanity.com/paper/2204.03498/) (2022)
- [Automatic Code Generation using Pre-Trained Language Models](https://mlsanity.com/paper/2102.10535/) (2021)
- [Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation](https://mlsanity.com/paper/2106.04447/) (2021)
- [Generalized Equivariance and Preferential Labeling for GNN Node Classification](https://mlsanity.com/paper/2102.11485/) (2021)
- [CodePAD: Sequence-based Code Generation with Pushdown Automaton](https://mlsanity.com/paper/2211.00818/) (2022)
- [CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking](https://mlsanity.com/paper/2402.11842/) (2024)
- [RANGER -- Repository-Level Agent for Graph-Enhanced Retrieval](https://mlsanity.com/paper/2509.25257/) (2025)

## Abstract (arXiv)

A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.