# Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

arXiv: https://arxiv.org/abs/2412.10079 · HTML: https://mlsanity.com/paper/2412.10079/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Multi-hop Question Answering](https://mlsanity.com/paper/2204.09140/) (2022)
- [Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models](https://mlsanity.com/paper/2310.07712/) (2023)
- [Measuring and Modifying Factual Knowledge in Large Language Models](https://mlsanity.com/paper/2306.06264/) (2023)
- [Robustifying Multi-hop QA through Pseudo-Evidentiality Training](https://mlsanity.com/paper/2107.03242/) (2021)
- [Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoning](https://mlsanity.com/paper/2311.08505/) (2023)
- [Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning](https://mlsanity.com/paper/2210.07138/) (2022)

## Built on by (most cited first, 13 of 13)

- [Rope to Nope and Back Again: A New Hybrid Attention Strategy](https://mlsanity.com/paper/2501.18795/) (2025)
- [Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation](https://mlsanity.com/paper/2508.15709/) (2025)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation](https://mlsanity.com/paper/2505.11754/) (2025)
- [Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck](https://mlsanity.com/paper/2601.12499/) (2026)
- [Modeling Contextual Passage Utility for Multihop Question Answering](https://mlsanity.com/paper/2512.06464/) (2025)
- [Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks](https://mlsanity.com/paper/2505.23843/) (2025)
- [Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding](https://mlsanity.com/paper/2507.14849/) (2025)
- [Use of Retrieval-Augmented Large Language Model Agent for Long-Form COVID-19 Fact-Checking](https://mlsanity.com/paper/2512.00007/) (2025)
- [UserGPT Technical Report](https://mlsanity.com/paper/2605.08766/) (2026)
- [IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking](https://mlsanity.com/paper/2606.09709/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering design](https://mlsanity.com/paper/2609.10132/) (2026)

## Abstract (arXiv)

Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.