mlsanityData to 7 Oct 2026

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

arXiv 2608.26004 ↗First posted 26 Aug 2026cs.AI cs.CLcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive δ-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches ≈ 90% of full-context accuracy on average, delivering 1.3--1.7× throughput speedups at 0.2--0.3× the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 78 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

56 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Optimizing Agentic Language Model Inference via Speculative Tool Calls17 Dec

Other references · 55

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
38 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  3. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  4. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  8. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  11. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  12. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 119
  13. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  14. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  15. 2025SGMem: Sentence Graph Memory for Long-Term Conversational Agentscited by 29
  16. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  17. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabulariescited by 20
  18. 2024Exploring User Retrieval Integration towards Large Language Models for Cross-Domain Sequential Recommendationcited by 19
  19. 2025Thought-Augmented Planning for LLM-Powered Interactive Recommender Agentcited by 17
  20. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  21. 2024Scaling New Frontiers: Insights into Large Recommendation Modelscited by 13
  22. 2025Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendationcited by 13
  23. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  24. 2025Killing Two Birds with One Stone: Unifying Retrieval and Ranking with a Single Generative Recommendation Modelcited by 12
  25. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  26. 2026Why Thinking Hurts: Diagnosing and Rectifying Linguistic Inertia in Large Language Models for Recommendationcited by 9
  27. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  28. 2026The Next Paradigm Is User-Centric Agent, Not Platform-Centric Servicecited by 8
  29. 2024Multi-granularity Interest Retrieval and Refinement Network for Long-Term User Behavior Modeling in CTR Predictioncited by 7
  30. 2025Adaptive Schema-aware Event Extraction with Retrieval-Augmented Generationcited by 7
  31. 2024Aligning Language Models Using Follow-up Likelihood as Reward Signalcited by 6
  32. 2026SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generationcited by 6
  33. 2026SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agentscited by 5
  34. 2026Can Recommender Systems Teach Themselves? A Recursive Self-Improving Framework with Fidelity Controlcited by 5
  35. 2026FuXi-Linear: Unleashing the Power of Linear Attention in Long-term Time-aware Sequential Recommendationcited by 5
  36. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  37. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  38. 2026SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utilitycited by 4

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026InsightChain: Optimized Chain-of-Insight Analytics for LLM-driven Data Visualizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
  2. 2026AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines
  3. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
  4. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
  5. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
  6. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  7. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
  8. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling