# Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge

arXiv: https://arxiv.org/abs/2412.08317 · HTML: https://mlsanity.com/paper/2412.08317/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 41 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [Multi-hop Question Answering](https://mlsanity.com/paper/2204.09140/) (2022)
- [Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention](https://mlsanity.com/paper/2112.03254/) (2021)
- [Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval](https://mlsanity.com/paper/2104.05883/) (2021)
- [Relational Graph Convolutional Neural Networks for Multihop Reasoning: A Comparative Study](https://mlsanity.com/paper/2210.06418/) (2022)

## Built on by (most cited first, 1 of 1)

- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)

## Abstract (arXiv)

We carry out a series of experiments to test large language models' multi-hop reasoning ability from three aspects: selecting and combining external knowledge, dealing with non-sequential reasoning tasks and generalising to data samples with larger numbers of hops. We test the GPT-3.5 model on four reasoning benchmarks with Chain-of-Thought prompting (and its variations). Our results reveal that despite the amazing performance achieved by large language models on various reasoning tasks, models still suffer from severe drawbacks which shows a large gap with humans.