mlsanityData to 7 Oct 2026

Do Transformers Need Deep Long-Range Memory

arXiv 2007.03356 ↗First posted 7 Jul 2020cs.LG cs.CL stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

Deep attention models have advanced the modelling of sequential data across many domains. For language modelling in particular, the Transformer-XL -- a Transformer augmented with a long-range memory of past activations -- has been shown to be state-of-the-art across a variety of well-studied benchmarks. The Transformer-XL incorporates a long-range memory at every layer of the network, which renders its state to be thousands of times larger than RNN predecessors. However it is unclear whether this is necessary. We perform a set of interventions to show that comparable performance can be obtained with 6X fewer long range memories and better performance can be obtained by limiting the range of attention in lower layers of the network.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 7; 2024: 4; 2025: 3; 2026: 2.

24 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  3. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  4. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  5. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  6. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  7. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  8. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  9. 2023Dissociating language and thought in large language modelscited by 27
  10. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  11. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 21
  12. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
12 more
  1. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  2. 2021Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trendscited by 15
  3. 2021Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCANcited by 14
  4. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  5. 2025What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?cited by 13
  6. 2023Zebra: Extending Context Window with Layerwise Grouped Local-Global Attentioncited by 6
  7. 2022Embedding Recycling for Language Modelscited by 5
  8. 2024Are queries and keys always relevant? A case study on Transformer wave functionscited by 5
  9. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  10. 2022iColoriT: Towards Propagating Local Hint to the Right Region in Interactive Colorization by Leveraging Vision Transformercited by 0
  11. 2026Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrencecited by 0
  12. 2026Transcript-Managed Transformers: Monotone Multi-Agent Collapse and Universality with Two Pop-Enabled Transcriptscited by 0