# CodePAD: Sequence-based Code Generation with Pushdown Automaton arXiv: https://arxiv.org/abs/2211.00818 ยท HTML: https://mlsanity.com/paper/2211.00818/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 40 references that are in the map. - First posted on arXiv 2 Nov 2022. ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015) - [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013) - [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017) - [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022) - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020) - [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021) - [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018) - [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016) - [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022) - [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016) - [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017) - [TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation](https://mlsanity.com/paper/1810.02720/) (2018) - [Latent Predictor Networks for Code Generation](https://mlsanity.com/paper/1603.06744/) (2016) - [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019) - [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019) - [JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation](https://mlsanity.com/paper/1910.02216/) (2019) - [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022) - [A Grammar-Based Structural CNN Decoder for Code Generation](https://mlsanity.com/paper/1811.06837/) (2018) - [Semantic Parsing with Dual Learning](https://mlsanity.com/paper/1907.05343/) (2019) - [Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation](https://mlsanity.com/paper/2208.10091/) (2022) - [Improving Tree-Structured Decoder Training for Code Generation via Mutual Learning](https://mlsanity.com/paper/2105.14796/) (2021) - [Exploring Dynamic Selection of Branch Expansion Orders for Code Generation](https://mlsanity.com/paper/2106.00261/) (2021) ## Built on by (most cited first, 7 of 7) - [Grammar-Aligned Decoding](https://mlsanity.com/paper/2405.21047/) (2024) - [Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective](https://mlsanity.com/paper/2506.05754/) (2025) - [Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation](https://mlsanity.com/paper/2506.03887/) (2025) - [Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars](https://mlsanity.com/paper/2602.00612/) (2026) - [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024) - [Refactoring with LLMs: Bridging Human Expertise and Machine Understanding](https://mlsanity.com/paper/2510.03914/) (2025) - [Bootstrapping Fuzzers for Compilers of Low-Resource Language Dialects Using Language Models](https://mlsanity.com/paper/2512.05887/) (2025) ## Abstract (arXiv) In the process of code generation, it is essential to guarantee the generated code satisfies grammar constraints of programming language (PL). However, neglecting grammar constraints is a fatal drawback of commonly used sequence-based code generation. In this paper, we devise a pushdown automaton (PDA)-based methodology to address this problem, exploiting the principle that PL is a subset of PDA recognizable language and code accepted by PDA is grammatical. Specifically, we construct a PDA module and design an algorithm to constrain the generation of sequence-based models to ensure grammatical correctness. Guided by this methodology, we further propose CodePAD, a sequence-based code generation framework equipped with a PDA module, to integrate the deduction of PDA into deep learning. Additionally, this framework can leverage states of PDA deduction (including state representation, state prediction task, and joint prediction with state) to assist models in learning PDA deduction. To comprehensively evaluate CodePAD, we construct a PDA for Python and conduct extensive experiments on four public benchmark datasets. CodePAD can leverage existing sequence-based models, and we show that it can achieve 100% grammatical correctness percentage on these benchmark datasets. Thus, it relatively improve 17% CodeBLEU on CONALA, 8% EM on DJANGO, and 15% CodeBLEU on JUICE-10K compared to base models. In addition, our method significantly enhances pre-trained models, e.g., CodeBLEU of CodeGen-350M improvement from 3.21 to 21.54 on MBPP in zero-shot setting.