mlsanityData to 7 Oct 2026

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

arXiv 2403.04797 ↗First posted 5 Mar 2024cs.CL cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

43 of 60 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Mistral 7Bcited by 3,065
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2024Mixtral of Expertscited by 1,785
31 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  9. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  10. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  12. 2023Augmenting Language Models with Long-Term Memorycited by 186
  13. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  14. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  15. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  16. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  17. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  18. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  19. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  20. 2024Transformers are Multi-State RNNscited by 120
  21. 2023Retrieval meets Long Context Large Language Modelscited by 114
  22. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  23. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  24. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  25. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  26. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  27. 2021On the Expressive Power of Self-Attention Matricescited by 44
  28. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  29. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  30. 2023DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologiescited by 14
  31. 2023Zebra: Extending Context Window with Layerwise Grouped Local-Global Attentioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 33; 2026: 24.

72 papers.

Sort:

By area:

  1. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  2. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  3. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  4. 2025On the Emergence of Position Bias in Transformerscited by 57
  5. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  6. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  7. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  8. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  9. 2025Lost at the Beginning of Reasoningcited by 15
  10. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  11. 2024Towards LifeSpan Cognitive Systemscited by 13
  12. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
38 more
  1. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compressioncited by 9
  2. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  3. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 8
  4. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  5. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  6. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  7. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  8. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  9. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  10. 2024ActionReasoningBench: Reasoning about Actions with and without Ramification Constraintscited by 4
  11. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
  12. 2024Quantifying Positional Biases in Text Embedding Modelscited by 4
  13. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  14. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  15. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  16. 2025Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigationcited by 3
  17. 2025CUB: Benchmarking Context Utilisation Techniques for Language Modelscited by 3
  18. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
  19. 2026A Structural Theory of Position Bias in Transformerscited by 3
  20. 2024Mitigate Position Bias in Large Language Models via Scaling a Single Dimensioncited by 2
  21. 2025RoToR: Towards More Reliable Responses for Order-Invariant Inputscited by 2
  22. 2025Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understandingcited by 2
  23. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
  24. 2026Making Theft Useless: Adulteration-Based Protection of Proprietary Knowledge Graphs in GraphRAG Systemscited by 2
  25. 2026ContextEvolve: Multi-Agent Context Compression for Systems Code Optimizationcited by 2
  26. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  27. 2024PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overheadcited by 1
  28. 2024Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Modelscited by 1
  29. 2025END: Early Noise Dropping for Efficient and Effective Context Denoisingcited by 1
  30. 2025Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibrationcited by 1
  31. 2025Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modelingcited by 1
  32. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understandingcited by 1
  33. 2025A novel language model for predicting serious adverse event results in clinical trials from their prospective registrationscited by 1
  34. 2025CCF: A Context Compression Framework for Efficient Long-Sequence Language Modelingcited by 1
  35. 2025Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learningcited by 1
  36. 2025Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contextscited by 1
  37. 2025Short-Context Dominance: How Much Local Context Natural Language Actually Needs?cited by 1
  38. 2025Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processingcited by 1

50 most cited of 72 All 72 (JSON)