# SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents

arXiv: https://arxiv.org/abs/2605.21965 · HTML: https://mlsanity.com/paper/2605.21965/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG](https://mlsanity.com/paper/2501.09136/) (2025)
- [Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting](https://mlsanity.com/paper/2407.08223/) (2024)
- [Chain-of-Retrieval Augmented Generation](https://mlsanity.com/paper/2501.14342/) (2025)
- [Decoding Speculative Decoding](https://mlsanity.com/paper/2402.01528/) (2024)
- [Speculative Actions: A Lossless Framework for Faster Agentic Systems](https://mlsanity.com/paper/2510.04371/) (2025)
- [Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies](https://mlsanity.com/paper/2502.05202/) (2025)
- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent](https://mlsanity.com/paper/2603.01152/) (2026)
- [DeepNote: Note-Centric Deep Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.08821/) (2024)
- [Knowing When to Ask -- Bridging Large Language Models and Data](https://mlsanity.com/paper/2409.13741/) (2024)
- [SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding](https://mlsanity.com/paper/2504.04104/) (2025)
- [PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding](https://mlsanity.com/paper/2505.01572/) (2025)
- [SPARC-RAG: Adaptive Sequential-Parallel Scaling with Context Management for Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.00083/) (2026)

## Built on by (most cited first, 2 of 2)

- [AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving](https://mlsanity.com/paper/2608.00881/) (2026)
- [ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation](https://mlsanity.com/paper/2610.15246/) (2026)

## Abstract (arXiv)

Large language models increasingly use external tools such as web search and document retrieval to solve information-intensive tasks. However, multi-hop tool use in complex tasks introduces substantial latency, since the model must repeatedly wait for tool observations before continuing. We study how to accelerate such trajectories without changing the final trajectory the model would have taken without acceleration, assuming access to faster but less reliable speculator tools. We develop a theoretical framework for lossless speculation in multi-hop tool-use settings, characterizing the optimal achievable latency gain. We propose SpecHop, a continuous speculation framework that maintains multiple speculative threads, verifies predicted observations asynchronously as target tool outputs arrive, commits correct branches, and rolls back incorrect ones. This preserves accuracy while reducing wall-clock latency. We show that SpecHop can approach oracle latency gains with enough active threads. Empirically, on retrieval-augmented multi-hop tasks, SpecHop closely matches theoretical predictions and reduces latency by up to 40% in some settings. Code: https://github.com/mehrdadsaberi/spechop