mlsanityData to 7 Oct 2026

Transformers are Multi-State RNNs

arXiv 2401.06104 ↗First posted 11 Jan 2024cs.CLcited by 120

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Transformers are considered conceptually different from the previous generation of state-of-the-art NLP models - recurrent neural networks (RNNs). In this work, we demonstrate that decoder-only transformers can in fact be conceptualized as unbounded multi-state RNNs - an RNN variant with unlimited hidden state size. We further show that transformers can be converted into textitbounded multi-state RNNs by fixing the size of their hidden state, effectively compressing their key-value cache. We introduce a novel, training-free compression policy - Token Omission Via Attention (TOVA). Our experiments with four long range tasks and several LLMs show that TOVA outperforms several baseline compression policies. Particularly, our results are nearly on par with the full model, using in some cases only frac18 of the original cache size, which translates to 4.8X higher throughput. Our results shed light on the connection between transformers and RNNs, and help mitigate one of LLMs' most painful computational bottlenecks - the size of their key-value cache. We publicly release our code at https://github.com/schwartz-lab-NLP/TOVA

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 64 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

36 of 64 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
24 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  10. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  11. 2021Pay Attention to MLPscited by 367
  12. 2021Random Feature Attentioncited by 350
  13. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  14. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  15. 2022Long Range Language Modeling via Gated State Spacescited by 265
  16. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  17. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  18. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  19. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  20. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  21. 2021ABC: Attention with Bounded-memory Controlcited by 36
  22. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  23. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 20
  24. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 47; 2026: 56.

120 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  3. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  4. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  5. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  9. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  10. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  11. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  12. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
38 more
  1. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  2. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  3. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  4. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  5. 2024TransformerFAM: Feedback attention is working memorycited by 21
  6. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  7. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  8. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
  9. 2025Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoningcited by 16
  10. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  11. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 15
  12. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  13. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  14. 2024Towards LifeSpan Cognitive Systemscited by 13
  15. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  16. 2024In-context KV-Cache Eviction for LLMs via Attention-Gatecited by 12
  17. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  18. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  19. 2026Learning to Evict from Key-Value Cachecited by 12
  20. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  21. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 11
  22. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  23. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  24. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 9
  25. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  26. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
  27. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  28. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  29. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  30. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inferencecited by 7
  31. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  32. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  33. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  34. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 6
  35. 2025Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformerscited by 6
  36. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 6
  37. 2025No Mean Feat: Simple, Strong Baselines for Context Compressioncited by 6
  38. 2026MKA: Memory-Keyed Attention for Efficient Long-Context Reasoningcited by 6

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Does Transformer Interpretability Transfer to RNNs?
  2. 2024Theoretical limitations of multi-layer Transformer
  3. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
  4. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  5. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
  6. 2026Revisiting the Shape Convention of Transformer Language Models
  7. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokens
  8. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving