Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
On the map: LLM reasoning → Knowledge- and tool-augmented reasoning
Abstract (arXiv)
We carry out a series of experiments to test large language models' multi-hop reasoning ability from three aspects: selecting and combining external knowledge, dealing with non-sequential reasoning tasks and generalising to data samples with larger numbers of hops. We test the GPT-3.5 model on four reasoning benchmarks with Chain-of-Thought prompting (and its variations). Our results reveal that despite the amazing performance achieved by large language models on various reasoning tasks, models still suffer from severe drawbacks which shows a large gap with humans.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 41 references that are in the map. First posted on arXiv 11 Dec 2024.
Builds on
10 of 41 references.
Closest ancestors · 1
Same direction, newest first
- 2022Relational Graph Convolutional Neural Networks for Multihop Reasoning: A Comparative Study12 Oct
Other references · 9
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
- 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
- 2021Entailment as Few-Shot Learnercited by 123
- 2022Multi-hop Question Answeringcited by 64
- 2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
- 2021Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrievalcited by 25
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?
- 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
- 2026Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language Models
- 2026Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
- 2024Making Long-Context Language Models Better Multi-Hop Reasoners
- 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration
- 2024Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning
- 2025Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis