# Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning

arXiv: https://arxiv.org/abs/2105.14796 · HTML: https://mlsanity.com/paper/2105.14796/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 19 of 19)

- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017)
- [TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation](https://mlsanity.com/paper/1810.02720/) (2018)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://mlsanity.com/paper/1907.04829/) (2019)
- [Latent Predictor Networks for Code Generation](https://mlsanity.com/paper/1603.06744/) (2016)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019)
- [Incorporating External Knowledge through Pre-training for Natural Language to Code Generation](https://mlsanity.com/paper/2004.09015/) (2020)
- [Program Synthesis and Semantic Parsing with Learned Code Idioms](https://mlsanity.com/paper/1906.10816/) (2019)
- [A Grammar-Based Structural CNN Decoder for Code Generation](https://mlsanity.com/paper/1811.06837/) (2018)
- [Asynchronous Bidirectional Decoding for Neural Machine Translation](https://mlsanity.com/paper/1801.05122/) (2018)
- [The University of Edinburgh's Neural MT Systems for WMT17](https://mlsanity.com/paper/1708.00726/) (2017)
- [Iterative Dual Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1912.07239/) (2019)
- [Improving Domain-Adapted Sentiment Classification by Deep Adversarial Mutual Learning](https://mlsanity.com/paper/2002.00119/) (2020)

## Built on by (most cited first, 10 of 10)

- [StructCoder: Structure-Aware Transformer for Code Generation](https://mlsanity.com/paper/2206.05239/) (2022)
- [EpiCoder: Encompassing Diversity and Complexity in Code Generation](https://mlsanity.com/paper/2501.04694/) (2025)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation](https://mlsanity.com/paper/2208.10091/) (2022)
- [Exploring Dynamic Selection of Branch Expansion Orders for Code Generation](https://mlsanity.com/paper/2106.00261/) (2021)
- [CodePAD: Sequence-based Code Generation with Pushdown Automaton](https://mlsanity.com/paper/2211.00818/) (2022)
- [A Survey of Deep Learning Models for Structural Code Understanding](https://mlsanity.com/paper/2205.01293/) (2022)
- [UniGenCoder: Merging Seq2Seq and Seq2Tree Paradigms for Unified Code Generation](https://mlsanity.com/paper/2502.12490/) (2025)
- [Antecedent Predictions Are More Important Than You Think: An Effective Method for Tree-Based Code Generation](https://mlsanity.com/paper/2208.09998/) (2022)
- [Guiding AMR Parsing with Reverse Graph Linearization](https://mlsanity.com/paper/2310.08860/) (2023)

## Abstract (arXiv)

Code generation aims to automatically generate a piece of code given an input natural language utterance. Currently, among dominant models, it is treated as a sequence-to-tree task, where a decoder outputs a sequence of actions corresponding to the pre-order traversal of an Abstract Syntax Tree. However, such a decoder only exploits the preorder traversal based preceding actions, which are insufficient to ensure correct action predictions. In this paper, we first throughly analyze the context modeling difference between neural code generation models with different traversals based decodings (preorder traversal vs breadth-first traversal), and then propose to introduce a mutual learning framework to jointly train these models. Under this framework, we continuously enhance both two models via mutual distillation, which involves synchronous executions of two one-to-one knowledge transfers at each training step. More specifically, we alternately choose one model as the student and the other as its teacher, and require the student to fit the training data and the action prediction distributions of its teacher. By doing so, both models can fully absorb the knowledge from each other and thus could be improved simultaneously. Experimental results and in-depth analysis on several benchmark datasets demonstrate the effectiveness of our approach. We release our code at https://github.com/DeepLearnXMU/CGML.