mlsanityData to 7 Oct 2026

Landmark Attention: Random-Access Infinite Context Length for Transformers

arXiv 2305.16300 ↗First posted 25 May 2023cs.CL cs.LGcited by 206

Not on the map. Why?

Cited by 33 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts. Prior approaches, such as recurrent memory or retrieval-based augmentation, have either compromised the random-access flexibility of attention (i.e., the capability to select any token in the entire context) or relied on separate mechanisms for relevant context retrieval, which may not be compatible with the model's attention. In this paper, we present a novel approach that allows access to the complete context while retaining random-access flexibility, closely resembling running attention on the entire context. Our method uses a landmark token to represent each block of the input and trains the attention to use it for selecting relevant blocks, enabling retrieval of blocks directly through the attention mechanism instead of by relying on a separate mechanism. Our approach seamlessly integrates with specialized data structures and the system's memory hierarchy, enabling processing of arbitrarily long context lengths. We demonstrate that our method can obtain comparable performance with Transformer-XL while significantly reducing the number of retrieved tokens in each step. Finally, we show that fine-tuning LLaMA 7B with our method successfully extends its context length capacity to over 32k tokens, allowing for inference at the context lengths of GPT-4. We release the implementation of landmark attention and the code to reproduce our experiments at https://github.com/epfml/landmark-attention/.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Builds on 37 of its 43 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

37 of 43 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Reformer: The Efficient Transformercited by 1,658
25 more
  1. 2020Rethinking Attention with Performerscited by 1,584
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2022Efficiently Scaling Transformer Inferencecited by 529
  10. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  13. 2019Revealing the Dark Secrets of BERTcited by 300
  14. 2022Memorizing Transformerscited by 219
  15. 2022Recurrent Memory Transformercited by 197
  16. 2019Adaptive Attention Span in Transformerscited by 148
  17. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  18. 2022A Length-Extrapolatable Transformercited by 123
  19. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  20. 2020Memory Transformercited by 74
  21. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  22. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  23. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  24. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  25. 2021∞-former: Infinite Memory Transformercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 90; 2025: 53; 2026: 38.

206 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,785
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
38 more
  1. 2023Improving Text Embeddings with Large Language Modelscited by 295
  2. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  3. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  4. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  6. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  8. 2023Efficient Large Language Models: A Surveycited by 183
  9. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  12. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  13. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  14. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  15. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  16. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  17. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  18. 2024Make Your LLM Fully Utilize the Contextcited by 118
  19. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  20. 2023Retrieval meets Long Context Large Language Modelscited by 114
  21. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  22. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  23. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  24. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  25. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  26. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  27. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  28. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  29. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  30. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  31. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  32. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  33. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  34. 2024Base of RoPE Bounds Context Lengthcited by 49
  35. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  36. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  37. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  38. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46

50 most cited of 206 All 206 (JSON)