# Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design arXiv: https://arxiv.org/abs/2511.20048 · HTML: https://mlsanity.com/paper/2511.20048/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 28 of its 36 references that are in the map. - First posted on arXiv 25 Nov 2025. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 28 of 28) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025) - [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025) - [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022) - [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021) - [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025) - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024) - [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023) - [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025) - [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025) - [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024) - [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025) - [AdaptThink: Reasoning Models Can Learn When to Think](https://mlsanity.com/paper/2505.13417/) (2025) - [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023) - [Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning](https://mlsanity.com/paper/2505.17813/) (2025) - [Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach](https://mlsanity.com/paper/2407.13101/) (2024) - [TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput](https://mlsanity.com/paper/2406.14066/) (2024) - [Speculative Actions: A Lossless Framework for Faster Agentic Systems](https://mlsanity.com/paper/2510.04371/) (2025) - [Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface](https://mlsanity.com/paper/2410.00079/) (2024) - [ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework](https://mlsanity.com/paper/2505.18105/) (2025) ## Built on by (most cited first, 8 of 8) - [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026) - [DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation](https://mlsanity.com/paper/2603.07416/) (2026) - [SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference](https://mlsanity.com/paper/2607.03333/) (2026) - [Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL](https://mlsanity.com/paper/2607.25816/) (2026) - [Second Thought: Reasoning in Parallel as LLM Agents Act and Observe](https://mlsanity.com/paper/2608.13667/) (2026) - [DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving](https://mlsanity.com/paper/2609.31047/) (2026) - [LEAP: Learning Efficient Action Proposals For LLM Agents](https://mlsanity.com/paper/2610.02670/) (2026) - [ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation](https://mlsanity.com/paper/2610.15246/) (2026) ## Abstract (arXiv) LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to 1.65× end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents.