mlsanityData to 14 Oct 2026

X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding

arXiv 2609.09166 ↗First posted 15 Jul 2026cs.CL cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

This paper investigates collaborative speculative decoding (CoSD), a distributed large language model (LLM) inference framework in which an on-device small language model (SLM) drafts candidate tokens and a server LLM verifies them. Existing CoSD methods assume a shared vocabulary between the SLM and the LLM and incur substantial communication load because residual resampling requires token distribution exchange between the user device and the edge server. To address these limitations, we propose cross-vocabulary CoSD (X-CoSD), a lossless and communication-efficient CoSD framework for heterogeneous SLM-LLM vocabularies. X-CoSD is built on hybrid resampling (HR), which splits residual resampling across the common-vocabulary region on the device and the LLM-only region on the server, so that distribution transmission is required only for the common-vocabulary region. We further propose X-CoSD-E, an enhanced variant based on server resampling with device verification (SR-DV), in which the server sends only replacement candidates sampled from the server LLM and their corresponding probabilities for local verification at the device. We prove that both X-CoSD and X-CoSD-E preserve the server LLM distribution, and experiments show that they significantly improve token generation speed while maintaining generation quality comparable to that of the server LLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 21 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

18 of 21 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding3 Jul
  2. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies31 Jan
  3. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  4. 2024Multi-Candidate Speculative Decoding12 Jan
  5. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2024Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Modelscited by 19
  9. 2025DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decodingcited by 15
  10. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 5
  11. 2023A Comprehensive Real-World Evaluation of 5G Improvements over 4G in Low- and Mid-Bandscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Fast Large Language Model Collaborative Decoding via Speculation
  2. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
  3. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  4. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  5. 2025Collaborative Speculative Inference for Efficient LLM Inference Serving
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  7. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
  8. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference