mlsanityData to 7 Oct 2026

Reformer: The Efficient Transformer

arXiv 2001.04451 ↗First posted 13 Jan 2020cs.LG cs.CL stat.MLcited by 1,658

Not on the map. Why?

Cited by 67 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O(L2) to O(Llog L), where L is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

Key facts. Cited by 1,658 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 243; 2022: 284; 2023: 344; 2024: 318; 2025: 274; 2026: 194.

1,658 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  11. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
38 more
  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  3. 2021Transformers in Vision: A Surveycited by 1,009
  4. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  5. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  6. 2021Multiscale Vision Transformerscited by 896
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2020Aligning AI With Shared Human Valuescited by 706
  12. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  13. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  14. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  15. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  16. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  17. 2022Efficiently Scaling Transformer Inferencecited by 529
  18. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  19. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  20. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  21. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  22. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  23. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  24. 2021A Survey of Transformerscited by 399
  25. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  26. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  27. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 387
  28. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  29. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  30. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  31. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  32. 2021Random Feature Attentioncited by 350
  33. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  34. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  35. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  36. 2021Efficient Attentions for Long Document Summarizationcited by 342
  37. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 335
  38. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 333

50 most cited of 1,658 All 1,658 (JSON)