mlsanityData to 7 Oct 2026

Compressive Transformers for Long-Range Sequence Modelling

arXiv 1911.05507 ↗First posted 13 Nov 2019cs.LG stat.MLcited by 699

Not on the map. Why?

Cited by 96 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Key facts. Cited by 699 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 61; 2022: 71; 2023: 102; 2024: 156; 2025: 142; 2026: 167.

699 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023A Comprehensive Overview of Large Language Modelscited by 639
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
38 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  9. 2021A Survey of Transformerscited by 399
  10. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  14. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  15. 2021Random Feature Attentioncited by 350
  16. 2023Adapting Language Models to Compress Contextscited by 350
  17. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  18. 2022AudioGen: Textually Guided Audio Generationcited by 292
  19. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  20. 2024Massive Activations in Large Language Modelscited by 277
  21. 2022Long Range Language Modeling via Gated State Spacescited by 265
  22. 2022Can language models learn from explanations in context?cited by 248
  23. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  24. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  25. 2022Memorizing Transformerscited by 219
  26. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  27. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  28. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  29. 2022Recurrent Memory Transformercited by 197
  30. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  31. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  32. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  33. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  34. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  35. 2022Transformer Quality in Linear Timecited by 189
  36. 2023Augmenting Language Models with Long-Term Memorycited by 186
  37. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  38. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181

50 most cited of 699 All 699 (JSON)