mlsanityData to 7 Oct 2026

When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

arXiv 2102.12459 ↗First posted 24 Feb 2021cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and Billion Word datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to top-performing Transformer models. For instance, our model achieves a state-of-the-art result on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

49 of 61 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
37 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2018Self-Attention with Relative Position Representationscited by 1,112
  4. 2020On Layer Normalization in the Transformer Architecturecited by 906
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2019Green AIcited by 620
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  13. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  14. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  15. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  16. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  17. 2019Understanding and Improving Layer Normalizationcited by 213
  18. 2020Understanding the Difficulty of Training Transformerscited by 210
  19. 2019Adaptive Attention Span in Transformerscited by 148
  20. 2020Fast Transformers with Clustered Attentioncited by 123
  21. 2019Augmenting Self-attention with Persistent Memorycited by 92
  22. 2016Quasi-Recurrent Neural Networkscited by 87
  23. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  24. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  25. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  26. 2016Efficient softmax approximation for GPUscited by 67
  27. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  28. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  29. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  30. 2019Optimizing Speech Recognition For The Edgecited by 32
  31. 2019Modeling Recurrence for Transformercited by 26
  32. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  33. 2016Optimizing Performance of Recurrent Neural Networks on GPUscited by 12
  34. 2020Addressing Some Limitations of Transformers with Feedback Memorycited by 8
  35. 2020Efficient Inference For Neural Machine Translationcited by 7
  36. 2019A Lightweight Recurrent Network for Sequence Modelingcited by 3
  37. 2020TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understandingcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 14; 2023: 6; 2024: 4; 2025: 5.

34 papers.

Sort:

By area:

  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2022Confident Adaptive Language Modelingcited by 297
  3. 2022Long Range Language Modeling via Gated State Spacescited by 265
  4. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  5. 2022Training Language Models with Memory Augmentationcited by 112
  6. 2022Block-Recurrent Transformerscited by 107
  7. 2021Finetuning Pretrained Transformers into RNNscited by 94
  8. 2021Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-designcited by 82
  9. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  10. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  11. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  12. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
22 more
  1. 2024DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Modelscited by 22
  2. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
  3. 2022Reprogramming Pretrained Language Models for Antibody Sequence Infillingcited by 20
  4. 2023Unsupervised Protein-Ligand Binding Energy Prediction via Neural Euler's Rotation Equationcited by 18
  5. 2022Antibody-Antigen Docking and Design via Hierarchical Equivariant Refinementcited by 15
  6. 2023Multi-Head State Space Model for Speech Recognitioncited by 14
  7. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  8. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10
  9. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  10. 2022Mukayese: Turkish NLP Strikes Backcited by 9
  11. 2023On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Researchcited by 7
  12. 2021SRU++: Pioneering Fast Recurrence with Attention for Speech Recognitioncited by 6
  13. 2024Cottention: Linear Transformers With Cosine Attentioncited by 5
  14. 2022Implicit N-grams Induced by Recurrencecited by 4
  15. 2022Simple Recurrence Improves Masked Language Modelscited by 4
  16. 2022Exploiting Expert Knowledge for Assigning Firms to Industries: A Novel Deep Learning Methodcited by 3
  17. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  18. 2025Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architecturescited by 1
  19. 2025ToxBench: A Binding Affinity Prediction Benchmark with AB-FEP-Calculated Labels for Human Estrogen Receptor Alphacited by 1
  20. 2022Circling Back to Recurrent Models of Languagecited by 0
  21. 2024SkipSNN: Efficiently Classifying Spike Trains with Event-attentioncited by 0
  22. 2025Energy-Based Models for Predicting Mutational Effects on Proteinscited by 0