mlsanityData to 7 Oct 2026

Effective Batching for Recurrent Neural Network Grammars

arXiv 2105.14822 ↗First posted 31 May 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

As a language model that integrates traditional symbolic operations and flexible neural representations, recurrent neural network grammars (RNNGs) have attracted great attention from both scientific and engineering perspectives. However, RNNGs are known to be harder to scale due to the difficulty of batched training. In this paper, we propose effective batching for RNNGs, where every operation is computed in parallel with tensors across multiple sentences. Our PyTorch implementation effectively employs a GPU and achieves x6 speedup compared to the existing C++ DyNet implementation with model-independent auto-batching. Moreover, our batched RNNG also accelerates inference and achieves x20-150 speedup for beam search depending on beam sizes. Finally, we evaluate syntactic generalization performance of the scaled RNNG against the LSTM baseline, based on the large training data of 100M tokens from English Wikipedia and the broad-coverage targeted syntactic evaluation benchmark. Our RNNG implementation is available at https://github.com/aistairc/rnng-pytorch/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

32 of 48 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  11. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  12. 2018Colorless green recurrent networks dream hierarchicallycited by 210
20 more
  1. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  2. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  3. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  4. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  5. 2016Recurrent Neural Network Grammarscited by 109
  6. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  7. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  8. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  9. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  10. 2017Effective Inference for Generative Neural Parsingcited by 24
  11. 2019Structural Supervision Improves Learning of Non-Local Grammatical Dependenciescited by 24
  12. 2020Syntactic Structure Distillation Pretraining For Bidirectional Encoderscited by 18
  13. 2017Learning to Parse and Translate Improves Neural Machine Translationcited by 16
  14. 2016What Do Recurrent Neural Network Grammars Learn About Syntax?cited by 15
  15. 2019Scalable Syntax-Aware Language Models Using Knowledge Distillationcited by 8
  16. 2020Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Modelscited by 8
  17. 2017On-the-fly Operation Batching in Dynamic Computation Graphscited by 7
  18. 2020An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Modelscited by 6
  19. 2018Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG inductioncited by 4
  20. 2019Parallelizable Stack Long Short-Term Memorycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 1; 2024: 3; 2025: 1; 2026: 2.

12 papers.

  1. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  2. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  3. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  4. 2021Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammarscited by 5
  5. 2023Psychometric Predictive Power of Large Language Modelscited by 4
  6. 2024Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervisioncited by 4
  7. 2025If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?cited by 4
  8. 2022Composition, Attention, or Both?cited by 1
  9. 2024Is Structure Dependence Shaped for Efficient Communication?: A Case Study on Coordinationcited by 1
  10. 2022Multilingual Syntax-aware Language Modeling through Dependency Tree Conversioncited by 0
  11. 2026Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesiscited by 0
  12. 2026GiLT: Augmenting Transformer Language Models with Dependency Graphscited by 0