mlsanityData to 7 Oct 2026

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

arXiv 2406.16008 ↗First posted 23 Jun 2024cs.CL cs.AI cs.LGcited by 97

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
28 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  13. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  15. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  16. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  17. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  18. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  19. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  20. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  21. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  22. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  23. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  24. 2022Parallel Context Windows for Large Language Modelscited by 81
  25. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  26. 2019Attention Interpretability Across NLP Taskscited by 71
  27. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  28. 2021On-the-Fly Attention Modulation for Neural Generationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 45; 2026: 39.

97 papers.

Sort:

By area:

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  2. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  3. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  4. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
  5. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  6. 2025FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generationcited by 27
  7. 2025AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generationcited by 19
  8. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  9. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 18
  10. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  11. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  12. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 17
38 more
  1. 2024Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generationcited by 15
  2. 2024Distance between Relevant Information Pieces Causes Bias in Long-Context LLMscited by 15
  3. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  4. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  5. 2025Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generationcited by 12
  6. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  7. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  8. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  9. 2024On the Loss of Context-awareness in General Instruction Fine-tuningcited by 10
  10. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  11. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  12. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  13. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  14. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  15. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 8
  16. 2025Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMscited by 8
  17. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problemscited by 7
  18. 2024Inference Scaling for Bridging Retrieval and Augmented Generationcited by 7
  19. 2025Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Datacited by 7
  20. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  21. 2024Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metriccited by 6
  22. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  23. 2026Spectral Attention Steering for Prompt Highlightingcited by 6
  24. 2025Unstructured Evidence Attribution for Long Context Query Focused Summarizationcited by 5
  25. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systemscited by 5
  26. 2025ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planningcited by 5
  27. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  28. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  29. 2026ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMscited by 5
  30. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 5
  31. 2025Stepwise Informativeness Search for Efficient and Effective LLM Reasoningcited by 4
  32. 2026Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibrationcited by 4
  33. 2025DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answeringcited by 3
  34. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  35. 2025ALAS: Transactional and Dynamic Multi-Agent LLM Planningcited by 3
  36. 2026The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agentscited by 3
  37. 2026Lost in the Middle at Birth: An Exact Theory of Transformer Position Biascited by 3
  38. 2025RoToR: Towards More Reliable Responses for Order-Invariant Inputscited by 2

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Positional Biases Shift as Inputs Approach Context Window Limits
  2. 2024Attention Instruction: Amplifying Attention in the Middle via Prompting
  3. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  4. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contexts
  5. 2026Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads
  6. 2026Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
  7. 2024Attention Heads of Large Language Models: A Survey
  8. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning