mlsanityData to 14 Oct 2026

Fast Large Language Model Collaborative Decoding via Speculation

arXiv 2502.01662 ↗First posted 1 Feb 2025cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) collaborative decoding techniques improve output quality by combining the outputs of multiple models at each generation step, but they incur high computational costs. In this paper, we introduce Collaborative decoding via Speculation (CoS), a novel framework that accelerates collaborative decoding without compromising performance. Inspired by Speculative Decoding--where a small proposal model generates tokens sequentially, and a larger target model verifies them in parallel, our approach builds on two key insights: (1) the verification distribution can be the combined distribution of both the proposal and target models, and (2) alternating each model as the proposer and verifier can further enhance efficiency. We generalize this method to collaboration among n models and theoretically prove that CoS is never slower than standard collaborative decoding, typically achieving faster speed. Extensive experiments demonstrate CoS is 1.11x-2.23x faster than standard collaborative decoding without compromising generation quality. Our code is available at https://github.com/Kamichanw/CoS/.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

33 of 37 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
21 more
  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  6. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  8. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 74
  9. 2024Decoding-time Realignment of Language Modelscited by 67
  10. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  11. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  12. 2023PaSS: Parallel Speculative Samplingcited by 49
  13. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  14. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  15. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 28
  16. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  17. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decodingcited by 22
  18. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inferencecited by 11
  19. 2024HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoningcited by 11
  20. 2023Speculative Contrastive Decodingcited by 8
  21. 2024Graph-Structured Speculative Decodingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  2. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 32
  3. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  4. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  5. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5
  6. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  7. 2025When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensemblingcited by 3
  8. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 3
  9. 2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
  10. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1
  11. 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmissioncited by 0
  12. 2026Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Target: From Imitation to Collaboration in Speculative Decoding
  2. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
  3. 2026X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding
  4. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  6. 2025Speculative Decoding: Performance or Illusion?
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2024Decoding Speculative Decoding