# Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

arXiv: https://arxiv.org/abs/2406.16008 · HTML: https://mlsanity.com/paper/2406.16008/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)

## Built on by (most cited first, 30 of 101)

- [Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG](https://mlsanity.com/paper/2410.05983/) (2024)
- [Sufficient Context: A New Lens on Retrieval Augmented Generation Systems](https://mlsanity.com/paper/2411.06037/) (2024)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planning](https://mlsanity.com/paper/2503.11951/) (2025)
- [Seed-Coder: Let the Code Model Curate Data for Itself](https://mlsanity.com/paper/2506.03524/) (2025)
- [FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.08938/) (2025)
- [Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering](https://mlsanity.com/paper/2505.12025/) (2025)
- [GATEAU: Selecting Influential Samples for Long Context Alignment](https://mlsanity.com/paper/2410.15633/) (2024)
- [AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10720/) (2025)
- [GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding](https://mlsanity.com/paper/2409.03258/) (2024)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management](https://mlsanity.com/paper/2508.04664/) (2025)
- [Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs](https://mlsanity.com/paper/2410.14641/) (2024)
- [Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.11598/) (2024)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.19754/) (2025)
- [Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?](https://mlsanity.com/paper/2411.05000/) (2024)
- [Do RAG Systems Really Suffer From Positional Bias?](https://mlsanity.com/paper/2505.15561/) (2025)
- [Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs](https://mlsanity.com/paper/2509.21361/) (2025)
- [On the Loss of Context-awareness in General Instruction Fine-tuning](https://mlsanity.com/paper/2411.02688/) (2024)
- [LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs](https://mlsanity.com/paper/2503.02502/) (2025)
- [Does quantization affect models' performance on long-context tasks?](https://mlsanity.com/paper/2505.20276/) (2025)
- [KeySG: Hierarchical Keyframe-Based 3D Scene Graphs](https://mlsanity.com/paper/2510.01049/) (2025)
- [AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Management](https://mlsanity.com/paper/2602.07398/) (2026)
- [Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models](https://mlsanity.com/paper/2408.10151/) (2024)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs](https://mlsanity.com/paper/2510.10276/) (2025)
- [Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data](https://mlsanity.com/paper/2505.02130/) (2025)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [Spectral Attention Steering for Prompt Highlighting](https://mlsanity.com/paper/2603.01281/) (2026)

## Abstract (arXiv)

Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.