mlsanityData to 7 Oct 2026

Block-Recurrent Transformers

arXiv 2203.07852 ↗First posted 11 Mar 2022cs.LG cs.AI cs.NEcited by 107

Not on the map. Why?

Abstract (arXiv)

We introduce the Block-Recurrent Transformer, which applies a transformer layer in a recurrent fashion along a sequence, and has linear complexity with respect to sequence length. Our recurrent cell operates on blocks of tokens rather than single tokens during training, and leverages parallel computation within a block in order to make efficient use of accelerator hardware. The cell itself is strikingly simple. It is merely a transformer layer: it uses self-attention and cross-attention to efficiently compute a recurrent function over a large set of state vectors and tokens. Our design was inspired in part by LSTM cells, and it uses LSTM-style gates, but it scales the typical LSTM cell up by several orders of magnitude. Our implementation of recurrence has the same cost in both computation time and parameter count as a conventional transformer layer, but offers dramatically improved perplexity in language modeling tasks over very long sequences. Our model out-performs a long-range Transformer XL baseline by a wide margin, while running twice as fast. We demonstrate its effectiveness on PG19 (books), arXiv papers, and GitHub source code. Our code has been released as open source.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
30 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2020Efficient Transformers: A Surveycited by 784
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  8. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  9. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  11. 2021Random Feature Attentioncited by 350
  12. 2015LSTM: A Search Space Odysseycited by 315
  13. 2020Query-Key Normalization for Transformerscited by 266
  14. 2022Transformer Quality in Linear Timecited by 189
  15. 2015Training Very Deep Networkscited by 172
  16. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  17. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  18. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  19. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  20. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  21. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  22. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  23. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  24. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  25. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  26. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  27. 2019R-Transformer: Recurrent Neural Network Enhanced Transformercited by 39
  28. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  29. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  30. 2021Staircase Attention for Recurrent Processing of Sequencescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 25; 2024: 29; 2025: 17; 2026: 23.

107 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2022Long Range Language Modeling via Gated State Spacescited by 265
  3. 2023Looped Transformers as Programmable Computerscited by 224
  4. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  9. 2022A Length-Extrapolatable Transformercited by 123
  10. 2025Test-Time Training Done Rightcited by 121
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  12. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
38 more
  1. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  2. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  3. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  4. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesiscited by 55
  5. 2024Language Repository for Long Video Understandingcited by 55
  6. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  7. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  8. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  9. 2022Temporally Consistent Transformers for Video Generationcited by 43
  10. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  11. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  12. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  13. 2023Learning to Reason and Memorize with Self-Notescited by 30
  14. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  15. 2022Token Turing Machinescited by 25
  16. 2023Block-State Transformerscited by 24
  17. 2023Efficient Attention via Control Variatescited by 23
  18. 2024TransformerFAM: Feedback attention is working memorycited by 21
  19. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  20. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  21. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  22. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
  23. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  24. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  25. 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
  26. 2022Exploring Transformers for Behavioural Biometrics: A Case Study in Gait Recognitioncited by 13
  27. 2025Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrievalcited by 13
  28. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  29. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  30. 2023Recurrent Attention Networks for Long-text Modelingcited by 12
  31. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
  32. 2023Large Language Model Programscited by 11
  33. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoningcited by 11
  34. 2026Addressable Memory for Video World Modelscited by 11
  35. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  36. 2023On the Long Range Abilities of Transformerscited by 10
  37. 2024Rethinking Transformers in Solving POMDPscited by 9
  38. 2024MELODI: Exploring Memory Compression for Long Contextscited by 9

50 most cited of 107 All 107 (JSON)