# AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

arXiv: https://arxiv.org/abs/2610.01108 · HTML: https://mlsanity.com/paper/2610.01108/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 30)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024)
- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale](https://mlsanity.com/paper/2409.16299/) (2024)
- [SAM Decoding: Speculative Decoding via Suffix Automaton](https://mlsanity.com/paper/2411.10666/) (2024)
- [SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications](https://mlsanity.com/paper/2411.04975/) (2024)
- [Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems](https://mlsanity.com/paper/2604.14228/) (2026)
- [Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models](https://mlsanity.com/paper/2405.04304/) (2024)
- [AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability](https://mlsanity.com/paper/2410.18351/) (2024)
- [Devstral: Fine-tuning Language Models for Coding Agent Applications](https://mlsanity.com/paper/2509.25193/) (2025)
- [Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation](https://mlsanity.com/paper/2411.18462/) (2024)
- [Speculative Decoding: Performance or Illusion?](https://mlsanity.com/paper/2601.11580/) (2025)
- [TeamBench: Evaluating Agent Coordination under Enforced Role Separation](https://mlsanity.com/paper/2605.07073/) (2026)
- [CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents](https://mlsanity.com/paper/2606.16824/) (2026)
- [FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification](https://mlsanity.com/paper/2502.17139/) (2025)

## Abstract (arXiv)

Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent pipelines: much of the reusable text is missing from their corpora or stored in a form that differs from what the agent emits, and their draft lengths ignore that accept length varies across agents and drifts over turns. We present AgSpec, a framework that supplies the corpus and draft-length policies that existing retrieval engines lack in coding-agent pipelines. AgSpec retrieves from session, workspace, and global corpora, retaining the ongoing session trajectory and indexing opened files in the agent's emission format. It bounds each agent's draft length with an offline-profiled cap and adapts the length online from verification feedback. On two repository-level multi-agent coding benchmarks, AgSpec outperforms five retrieval-based drafters and EAGLE-3 in most evaluated settings, raising generation throughput over autoregressive decoding up to 4.37× at batch size 1 and 4.76× at batch size 16. AgSpec also remains effective on benchmarks without a repository or a multi-agent pipeline, showing that its gains generalize to coding agents broadly.