# Evaluating the Sensitivity of LLMs to Prior Context

arXiv: https://arxiv.org/abs/2506.00069 · HTML: https://mlsanity.com/paper/2506.00069/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue](https://mlsanity.com/paper/2308.03549/) (2023)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://mlsanity.com/paper/2410.15553/) (2024)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.16745/) (2024)
- [Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://mlsanity.com/paper/2402.02244/) (2024)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Large Language Models in Education: Vision and Opportunities](https://mlsanity.com/paper/2311.13160/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction](https://mlsanity.com/paper/2603.04783/) (2026)
- [The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior](https://mlsanity.com/paper/2509.19364/) (2025)
- [Mitigating Conversational Inertia in Multi-Turn Agents](https://mlsanity.com/paper/2602.03664/) (2026)
- [Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems](https://mlsanity.com/paper/2603.03111/) (2026)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues](https://mlsanity.com/paper/2509.17694/) (2025)
- [AMEL: Accumulated Message Effects on LLM Judgments](https://mlsanity.com/paper/2605.22714/) (2026)
- [AI Guardrail Survival under Single-Cycle Agentic Self-Summarization](https://mlsanity.com/paper/2608.11392/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly deployed in multi-turn dialogue and other sustained interactive scenarios, it is essential to understand how extended context affects their performance. Popular benchmarks, focusing primarily on single-turn question answering (QA) tasks, fail to capture the effects of multi-turn exchanges. To address this gap, we introduce a novel set of benchmarks that systematically vary the volume and nature of prior context. We evaluate multiple conventional LLMs, including GPT, Claude, and Gemini, across these benchmarks to measure their sensitivity to contextual variations. Our findings reveal that LLM performance on multiple-choice questions can degrade dramatically in multi-turn interactions, with performance drops as large as 73% for certain models. Even highly capable models such as GPT-4o exhibit up to a 32% decrease in accuracy. Notably, the relative performance of larger versus smaller models is not always predictable. Moreover, the strategic placement of the task description within the context can substantially mitigate performance drops, improving the accuracy by as much as a factor of 3.5. These findings underscore the need for robust strategies to design, evaluate, and mitigate context-related sensitivity in LLMs.