mlsanityData to 7 Oct 2026

Unlimiformer: Long-Range Transformers with Unlimited Length Input

arXiv 2305.01625 ↗First posted 2 May 2023cs.CLcited by 127

Not on the map. Why?

Abstract (arXiv)

Since the proposal of transformers, these models have been limited to bounded input lengths, because of their need to attend to every token in the input. In this work, we propose Unlimiformer: a general approach that wraps any existing pretrained encoder-decoder transformer, and offloads the cross-attention computation to a single k-nearest-neighbor (kNN) index, while the returned kNN distances are the attention dot-product scores. This kNN index can be kept on either the GPU or CPU memory and queried in sub-linear time; this way, we can index practically unlimited input sequences, while every attention head in every decoder layer retrieves its top-k keys, instead of attending to every key. We evaluate Unlimiformer on several long-document and book-summarization benchmarks, showing that it can process even 500k token-long inputs from the BookSum dataset, without any input truncation at test time. We demonstrate that Unlimiformer improves pretrained models such as BART and Longformer by extending them to unlimited inputs without additional learned weights and without modifying their code. We make our code and models publicly available at https://github.com/abertsch72/unlimiformer .

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

48 of 56 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
36 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2020Efficient Transformers: A Surveycited by 784
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  14. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  15. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  16. 2021Efficient Attentions for Long Document Summarizationcited by 342
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2022Memorizing Transformerscited by 219
  19. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  20. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  21. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  22. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  23. 2020Fast Transformers with Clustered Attentioncited by 123
  24. 2022Training Language Models with Memory Augmentationcited by 112
  25. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  26. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  27. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  28. 2018Content Selection in Deep Learning Models of Summarizationcited by 64
  29. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  30. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  31. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  32. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  33. 2021Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversationscited by 42
  34. 2021Long Document Summarization in a Low Resource Setting using Pretrained Language Modelscited by 24
  35. 2020Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encodingcited by 19
  36. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 60; 2025: 32; 2026: 5.

127 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  3. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  7. 2023Generating Images with Multimodal Language Modelscited by 259
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  10. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  11. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  12. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
38 more
  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  7. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  8. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  10. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  11. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  12. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  13. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  14. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  15. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  16. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  17. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  18. 2023Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Textcited by 48
  19. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  20. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  21. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  22. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 33
  23. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  24. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  25. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  26. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  27. 2025Cognitive Memory in Large Language Modelscited by 28
  28. 2024What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detectioncited by 24
  29. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMscited by 24
  30. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  31. 2024Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlesslycited by 23
  32. 2023ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Gamescited by 22
  33. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  34. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  35. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  36. 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
  37. 2024Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learningcited by 17
  38. 2023Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPTcited by 16

50 most cited of 127 All 127 (JSON)