mlsanityData to 14 Oct 2026

Making Long-Context Language Models Better Multi-Hop Reasoners

arXiv 2408.03246 ↗First posted 6 Aug 2024cs.CLcited by 26

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 41 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

21 of 41 references.

References · 21

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  11. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
9 more
  1. 2022Teaching language models to support answers with verified quotescited by 292
  2. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  3. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  4. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  5. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  6. 2022Faithful Reasoning Using Large Language Modelscited by 130
  7. 2023A Survey of Large Language Models Attributioncited by 71
  8. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  9. 2023End-to-End Beam Retrieval for Multi-Hop Question Answeringcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 9.

26 papers.

Sort:

By area:

  1. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  2. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  3. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  4. 2025Credible Plan-Driven RAG Method for Multi-Hop Question Answeringcited by 12
  5. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  6. 2025Positional Biases Shift as Inputs Approach Context Window Limitscited by 9
  7. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  8. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contextscited by 8
  9. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  10. 2025FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledgecited by 6
  11. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrievalcited by 5
  12. 2025CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capabilitycited by 4
14 more
  1. 2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3
  2. 2026Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Modelscited by 3
  3. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 3
  4. 2026Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneckcited by 3
  5. 2025MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Modelscited by 2
  6. 2025Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queriescited by 2
  7. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  8. 2026A Decomposition Perspective to Long-context Reasoning for LLMscited by 1
  9. 2025Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformerscited by 0
  10. 2025C2-Cite: Contextual-Aware Citation Generation for Attributed Large Language Modelscited by 0
  11. 2026Document Reconstruction Unlocks Scalable Long-Context RLVRcited by 0
  12. 2026How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superpositioncited by 0
  13. 2026StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learningcited by 0
  14. 2026TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context
  2. 2024Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
  3. 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?
  4. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA
  5. 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis
  6. 2024Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning
  7. 2026Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
  8. 2026Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models