mlsanityData to 14 Oct 2026

Long Range Arena: A Benchmark for Efficient Transformers

arXiv 2011.04006 ↗First posted 8 Nov 2020cs.LG cs.AI cs.CLcited by 700

Not on the map. Why?

Cited by 60 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from 1K to 16K tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

Key facts. Cited by 700 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 93; 2022: 102; 2023: 141; 2024: 172; 2025: 122; 2026: 70.

700 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  12. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
38 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  2. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  3. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  5. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  7. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 419
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  10. 2021Random Feature Attentioncited by 354
  11. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  12. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  13. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  14. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  15. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  16. 2022Long Range Language Modeling via Gated State Spacescited by 265
  17. 2022Long Range Graph Benchmarkcited by 263
  18. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  19. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  20. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  21. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  22. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  23. 2022Memorizing Transformerscited by 226
  24. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  25. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  26. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  27. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  28. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  29. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  30. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  31. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  32. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  33. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  34. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  35. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  36. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  37. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  38. 2021A Dataset for Answering Time-Sensitive Questionscited by 154

50 most cited of 700 All 700 (JSON)