# Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration arXiv: https://arxiv.org/abs/2412.00061 ยท HTML: https://mlsanity.com/paper/2412.00061/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. - Builds on 21 of its 28 references that are in the map. - First posted on arXiv 25 Nov 2024. ## Placed in - Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/) ## Builds on (most cited first, 21 of 21) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024) - [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018) - [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024) - [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022) - [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023) - [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023) - [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023) - [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019) - [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024) - [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023) - [Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding](https://mlsanity.com/paper/2106.04970/) (2021) - [SPEED: Speculative Pipelined Execution for Efficient Decoding](https://mlsanity.com/paper/2310.12072/) (2023) - [Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding](https://mlsanity.com/paper/2307.05908/) (2023) ## Built on by (most cited first, 15 of 15) - [Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance](https://mlsanity.com/paper/2507.22424/) (2025) - [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025) - [Utility-Driven Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2506.20675/) (2025) - [KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization](https://mlsanity.com/paper/2505.16162/) (2025) - [Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective](https://mlsanity.com/paper/2602.00286/) (2026) - [KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models](https://mlsanity.com/paper/2603.01581/) (2026) - [HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness](https://mlsanity.com/paper/2603.17573/) (2026) - [Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs](https://mlsanity.com/paper/2506.05387/) (2025) - [DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting](https://mlsanity.com/paper/2607.07409/) (2026) - [Drax: Speech Recognition with Discrete Flow Matching](https://mlsanity.com/paper/2510.04162/) (2025) - [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026) - [SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration](https://mlsanity.com/paper/2604.12247/) (2026) - [READER: Retrieval-Assisted Drafter for Efficient LLM Inference](https://mlsanity.com/paper/2508.09072/) (2025) - [Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection](https://mlsanity.com/paper/2607.05148/) (2026) - [Contextual Correction of Parallel Draftsfor Speculative Decoding](https://mlsanity.com/paper/2610.12570/) (2026) ## Abstract (arXiv) Inference acceleration of large language models (LLMs) has been put forward in many application scenarios and speculative decoding has shown its advantage in addressing inference acceleration. Speculative decoding usually introduces a draft model to assist the base LLM where the draft model produces drafts and the base LLM verifies the draft for acceptance or rejection. In this framework, the final inference speed is decided by the decoding speed of the draft model and the acceptance rate of the draft provided by the draft model. Currently the widely used draft models usually generate draft tokens for the next several positions in a non-autoregressive way without considering the correlations between draft tokens. Therefore, it has a high decoding speed but an unsatisfactory acceptance rate. In this paper, we focus on how to improve the performance of the draft model and aim to accelerate inference via a high acceptance rate. To this end, we propose a CTC-based draft model which strengthens the correlations between draft tokens during the draft phase, thereby generating higher-quality draft candidate sequences. Experiment results show that compared to strong baselines, the proposed method can achieve a higher acceptance rate and hence a faster inference speed.