mlsanityData to 7 Oct 2026

Do Long-Range Language Models Actually Use Long-Range Context?

arXiv 2109.09115 ↗First posted 19 Sep 2021cs.CLcited by 92

Not on the map. Why?

Abstract (arXiv)

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the Routing Transformer, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

33 of 46 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
21 more
  1. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  5. 2014Memory Networkscited by 354
  6. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  7. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  8. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  9. 2019Adaptively Sparse Transformerscited by 154
  10. 2019Adaptive Attention Span in Transformerscited by 148
  11. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  12. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  13. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  14. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  15. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  16. 2020Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translationcited by 27
  17. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  18. 2020Local Self-Attention over Long Text for Efficient Document Retrievalcited by 18
  19. 2016Prediction with a Short Memorycited by 13
  20. 2015Larger-Context Language Modellingcited by 10
  21. 2015Document Context Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 13; 2023: 25; 2024: 29; 2025: 15; 2026: 7.

92 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  5. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2022Memorizing Transformerscited by 219
  8. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  10. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  11. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  12. 2023Personality Traits in Large Language Modelscited by 166
38 more
  1. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  3. 2024Make Your LLM Fully Utilize the Contextcited by 118
  4. 2022Block-Recurrent Transformerscited by 107
  5. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  6. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  7. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  8. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  9. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  10. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  11. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  12. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  13. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  14. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  15. 2024Towards Understanding the Working Mechanism of Text-to-Image Diffusion Modelcited by 48
  16. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  17. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  18. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  19. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  20. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  21. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  22. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  23. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  24. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
  25. 2024Long Context RAG Performance of Large Language Modelscited by 23
  26. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  27. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  28. 2022Model Criticism for Long-Form Text Generationcited by 21
  29. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  30. 2024Extending LLMs' Context Window with 100 Samplescited by 20
  31. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  32. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  33. 2023S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Modelscited by 18
  34. 2023BatchPrompt: Accomplish more with lesscited by 16
  35. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16
  36. 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15
  37. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 15
  38. 2024Mamba-PTQ: Outlier Channels in Recurrent Large Language Modelscited by 15

50 most cited of 92 All 92 (JSON)