mlsanityData to 7 Oct 2026

Circling Back to Recurrent Models of Language

arXiv 2211.01848 ↗First posted 3 Nov 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Just because some purely recurrent models suffer from being hard to optimize and inefficient on today's hardware, they are not necessarily bad models of language. We demonstrate this by the extent to which these models can still be improved by a combination of a slightly better recurrent cell, architecture, objective, as well as optimization. In the process, we establish a new state of the art for language modelling on small datasets and on Enwik8 with dynamic evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

23 of 32 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  9. 2015Importance Weighted Autoencoderscited by 454
  10. 2016Using the Output Embedding to Improve Language Modelscited by 266
  11. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  12. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
11 more
  1. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  2. 2020The Hardware Lotterycited by 160
  3. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  4. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  5. 2017Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimizationcited by 56
  6. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  7. 2018Can recurrent neural networks warp time?cited by 53
  8. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  9. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  10. 2018Pushing the bounds of dropoutcited by 2
  11. 2022Two-Tailed Averaging: Anytime, Adaptive, Once-in-a-While Optimal Weight Averaging for Better Generalizationcited by 1

Built on by

Not cited yet.