mlsanityData to 7 Oct 2026

Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

arXiv 2412.10079 ↗First posted 13 Dec 2024cs.CLcited by 13

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 35 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

17 of 35 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  7. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  9. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  10. 2024Make Your LLM Fully Utilize the Contextcited by 118
  11. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  12. 2022Multi-hop Question Answeringcited by 64
5 more
  1. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  2. 2023Measuring and Modifying Factual Knowledge in Large Language Modelscited by 16
  3. 2021Robustifying Multi-hop QA through Pseudo-Evidentiality Trainingcited by 13
  4. 2023Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoningcited by 9
  5. 2022Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  2. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  3. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  4. 2025Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutationcited by 3
  5. 2026Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneckcited by 3
  6. 2025Modeling Contextual Passage Utility for Multihop Question Answeringcited by 2
  7. 2025Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Taskscited by 1
  8. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understandingcited by 1
  9. 2025Use of Retrieval-Augmented Large Language Model Agent for Long-Form COVID-19 Fact-Checkingcited by 0
  10. 2026UserGPT Technical Reportcited by 0
  11. 2026IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinkingcited by 0
  12. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
1 more
  1. 2026Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context
  2. 2026Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering
  3. 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis
  4. 2026Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends
  5. 2024Making Long-Context Language Models Better Multi-Hop Reasoners
  6. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  7. 2026BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
  8. 2026Do Language Models Reason Across Languages?