# Effective Batching for Recurrent Neural Network Grammars

arXiv: https://arxiv.org/abs/2105.14822 · HTML: https://mlsanity.com/paper/2105.14822/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019)
- [Transition-Based Dependency Parsing with Stack Long Short-Term Memory](https://mlsanity.com/paper/1505.08075/) (2015)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [Effective Inference for Generative Neural Parsing](https://mlsanity.com/paper/1707.08976/) (2017)
- [Structural Supervision Improves Learning of Non-Local Grammatical Dependencies](https://mlsanity.com/paper/1903.00943/) (2019)
- [Syntactic Structure Distillation Pretraining For Bidirectional Encoders](https://mlsanity.com/paper/2005.13482/) (2020)
- [Learning to Parse and Translate Improves Neural Machine Translation](https://mlsanity.com/paper/1702.03525/) (2017)
- [What Do Recurrent Neural Network Grammars Learn About Syntax?](https://mlsanity.com/paper/1611.05774/) (2016)
- [Scalable Syntax-Aware Language Models Using Knowledge Distillation](https://mlsanity.com/paper/1906.06438/) (2019)
- [Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2010.05725/) (2020)
- [On-the-fly Operation Batching in Dynamic Computation Graphs](https://mlsanity.com/paper/1705.07860/) (2017)
- [An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models](https://mlsanity.com/paper/2004.02451/) (2020)

## Built on by (most cited first, 12 of 12)

- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars](https://mlsanity.com/paper/2109.04939/) (2021)
- [Psychometric Predictive Power of Large Language Models](https://mlsanity.com/paper/2311.07484/) (2023)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?](https://mlsanity.com/paper/2502.11469/) (2025)
- [Composition, Attention, or Both?](https://mlsanity.com/paper/2210.12958/) (2022)
- [Is Structure Dependence Shaped for Efficient Communication?: A Case Study on Coordination](https://mlsanity.com/paper/2410.10556/) (2024)
- [GiLT: Augmenting Transformer Language Models with Dependency Graphs](https://mlsanity.com/paper/2605.15562/) (2026)
- [Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion](https://mlsanity.com/paper/2204.08644/) (2022)
- [Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis](https://mlsanity.com/paper/2605.15440/) (2026)

## Abstract (arXiv)

As a language model that integrates traditional symbolic operations and flexible neural representations, recurrent neural network grammars (RNNGs) have attracted great attention from both scientific and engineering perspectives. However, RNNGs are known to be harder to scale due to the difficulty of batched training. In this paper, we propose effective batching for RNNGs, where every operation is computed in parallel with tensors across multiple sentences. Our PyTorch implementation effectively employs a GPU and achieves x6 speedup compared to the existing C++ DyNet implementation with model-independent auto-batching. Moreover, our batched RNNG also accelerates inference and achieves x20-150 speedup for beam search depending on beam sizes. Finally, we evaluate syntactic generalization performance of the scaled RNNG against the LSTM baseline, based on the large training data of 100M tokens from English Wikipedia and the broad-coverage targeted syntactic evaluation benchmark. Our RNNG implementation is available at https://github.com/aistairc/rnng-pytorch/.