Data to 7 Oct 2026

Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

arXiv 1805.04623 ↗First posted 12 May 2018cs.CLcited by 113

Not on the map. Why?

Abstract (arXiv)

We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the distant history. The model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context (beyond 50 tokens), suggesting the distant past is modeled only as a rough semantic field or topic. We further find that the neural caching model (Grave et al., 2017b) especially helps the LSTM to copy words from within this distant context. Overall, our analysis not only provides a better understanding of how neural LMs use their context, but also sheds light on recent success from cache-based models.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 25; 2023: 28; 2024: 14; 2025: 8; 2026: 12.

113 papers.

Sort:

By area:

2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
2022Locally Typical Samplingcited by 108
2022Block-Recurrent Transformerscited by 107
2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74

38 more

2021A Practical Survey on Faster and Lighter Transformerscited by 73
2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
2022Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Versioncited by 59
2024Language Repository for Long Video Understandingcited by 55
2021What Context Features Can Transformer Language Models Use?cited by 49
2026MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agentscited by 49
2021AutoCTS: Automated Correlated Time Series Forecasting -- Extended Versioncited by 48
2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
2022How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Modelscited by 43
2021Approximating How Single Head Attention Learnscited by 38
2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
2022On the Explainability of Natural Language Processing Deep Modelscited by 37
2022Context Limitations Make Neural Language Models More Human-Likecited by 32
2023Conformal Nucleus Samplingcited by 32
2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
2022A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generationcited by 25
2023Block-State Transformerscited by 24
2021A surprisal--duration trade-off across and within the world's languagescited by 22
2023Stealing the Decoding Algorithms of Language Modelscited by 19
2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16
2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13
2021On-the-Fly Attention Modulation for Neural Generationcited by 12
2021Adapting Long Context NLM for ASR Rescoring in Conversational Agentscited by 12
2023Toward a Theory of Causation for Interpreting Neural Code Modelscited by 12
2022Dynamic Global Memory for Document-level Argument Extractioncited by 11
2021Revisiting Simple Neural Probabilistic Language Modelscited by 10
2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
2021Studying word order through iterative shufflingcited by 9
2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
2021Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networkscited by 8
2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8
2021Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognitioncited by 7
2021ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modelingcited by 7

50 most cited of 113 All 113 (JSON)