mlsanityData to 7 Oct 2026

What Context Features Can Transformer Language Models Use?

arXiv 2106.08367 ↗First posted 15 Jun 2021cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations -- including shuffling word order within sentences and deleting all words other than nouns -- remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2019Attention is not Explanationcited by 916
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
14 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  3. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  4. 2016Understanding Neural Networks through Representation Erasurecited by 223
  5. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  6. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  7. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  8. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  9. 2018Language GANs Falling Shortcited by 97
  10. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  11. 2016Tracking the World State with Recurrent Entity Networkscited by 51
  12. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  13. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  14. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 13; 2024: 6; 2025: 4; 2026: 3.

49 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  3. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  5. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  6. 2022Can language models learn from explanations in context?cited by 248
  7. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  8. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  11. 2022HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequencescited by 57
  12. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
37 more
  1. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  2. 2022How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Modelscited by 43
  3. 2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
  4. 2022Identifying and Manipulating the Personality Traits of Language Modelscited by 40
  5. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  6. 2022Context Limitations Make Neural Language Models More Human-Likecited by 32
  7. 2024Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Braincited by 32
  8. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  9. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  10. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  11. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  12. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  13. 2023Bring Your Own Data! Self-Supervised Evaluation for Large Language Modelscited by 17
  14. 2021Local Structure Matters Most: Perturbation Study in NLUcited by 10
  15. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  16. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  17. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8
  18. 2022Black-box language model explanation by context length probingcited by 7
  19. 2023Revisiting Entropy Rate Constancy in Textcited by 7
  20. 2023Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewritingcited by 6
  21. 2024A Human-Computer Collaborative Tool for Training a Single Large Language Model Agent into a Network through Few Examplescited by 5
  22. 2024Benchmark Granularity and Model Robustness for Image-Text Retrievalcited by 5
  23. 2025Effective Context in Neural Speech Modelscited by 5
  24. 2021Quantifying the Task-Specific Information in Text-Based Classificationscited by 3
  25. 2023Revisiting Topic-Guided Language Modelscited by 3
  26. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  27. 2023MTCue: Learning Zero-Shot Control of Extra-Textual Attributes by Leveraging Unstructured Context in Neural Machine Translationcited by 2
  28. 2025Radar: Fast Long-Context Decoding for Any Transformercited by 2
  29. 2022Local Structure Matters Most in Most Languagescited by 1
  30. 2023Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialoguecited by 1
  31. 2024LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBotcited by 1
  32. 2026Right Reset: Chunking by Prefix Removalcited by 1
  33. 2024DispaRisk: Auditing Fairness Through Usable Informationcited by 0
  34. 2025What's in a prompt? Language models encode literary style in prompt embeddingscited by 0
  35. 2025Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discriminationcited by 0
  36. 2026A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMscited by 0
  37. 2026dQwen3.5: Hybrid-Attention Diffusion Language Modelscited by 0