Sentence-Level Contextual Entrainment in Large Language Models
Not on the map. Why?
Abstract (arXiv)
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
22 of 35 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
10 more
- 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
- 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
- 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
- 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
- 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
- 2025On the Alignment of Large Language Models with Global Human Opinioncited by 13
- 2024Robust Evaluation Measures for Evaluating Social Biases in Masked Language Modelscited by 5
- 2023Mining Effective Features Using Quantum Entropy for Humor Recognitioncited by 4
- 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMscited by 3
- 2026Better and Worse with Scale: How Contextual Entrainment Diverges with Model Sizecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.