# Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

arXiv: https://arxiv.org/abs/2403.04797 · HTML: https://mlsanity.com/paper/2403.04797/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Inductive Biases and Variable Creation in Self-Attention Mechanisms](https://mlsanity.com/paper/2110.10090/) (2021)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)

## Built on by (most cited first, 30 of 72)

- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference](https://mlsanity.com/paper/2502.20766/) (2025)
- [Why Does the Effective Context Length of LLMs Fall Short?](https://mlsanity.com/paper/2410.18745/) (2024)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell](https://mlsanity.com/paper/2406.14673/) (2024)
- [DAPE: Data-Adaptive Positional Encoding for Length Extrapolation](https://mlsanity.com/paper/2405.14722/) (2024)
- [From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data](https://mlsanity.com/paper/2406.19292/) (2024)
- [Mixture of In-Context Experts Enhance LLMs' Long Context Awareness](https://mlsanity.com/paper/2406.19598/) (2024)
- [Lost at the Beginning of Reasoning](https://mlsanity.com/paper/2506.22058/) (2025)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [DAPE V2: Process Attention Score as Feature Map for Length Extrapolation](https://mlsanity.com/paper/2410.04798/) (2024)
- [SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression](https://mlsanity.com/paper/2507.05633/) (2025)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs](https://mlsanity.com/paper/2510.10276/) (2025)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Attention Instruction: Amplifying Attention in the Middle via Prompting](https://mlsanity.com/paper/2406.17095/) (2024)
- [Single-Pass Document Scanning for Question Answering](https://mlsanity.com/paper/2504.03101/) (2025)
- [Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models](https://mlsanity.com/paper/2506.07334/) (2025)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [Hearing the Order: Investigating Position Bias in Large Audio-Language Models](https://mlsanity.com/paper/2510.00628/) (2025)
- [Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.02993/) (2026)
- [A Structural Theory of Position Bias in Transformers](https://mlsanity.com/paper/2602.16837/) (2026)
- [ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints](https://mlsanity.com/paper/2406.04046/) (2024)
- [CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling](https://mlsanity.com/paper/2406.12018/) (2024)
- [Quantifying Positional Biases in Text Embedding Models](https://mlsanity.com/paper/2412.15241/) (2024)
- [CUB: Benchmarking Context Utilisation Techniques for Language Models](https://mlsanity.com/paper/2505.16518/) (2025)
- [Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents](https://mlsanity.com/paper/2602.01405/) (2026)
- [The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving](https://mlsanity.com/paper/2405.11299/) (2024)
- [Mitigate Position Bias in Large Language Models via Scaling a Single Dimension](https://mlsanity.com/paper/2406.02536/) (2024)

## Abstract (arXiv)

This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.