# Training Domain Draft Models for Speculative Decoding: Best Practices and Insights

arXiv: https://arxiv.org/abs/2503.07807 · HTML: https://mlsanity.com/paper/2503.07807/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 8 of 8)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Composition of Experts: A Modular Compound AI System Leveraging Large Language Models](https://mlsanity.com/paper/2412.01868/) (2024)

## Built on by (most cited first, 10 of 10)

- [DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding](https://mlsanity.com/paper/2504.05598/) (2025)
- [Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner](https://mlsanity.com/paper/2508.15044/) (2025)
- [Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference](https://mlsanity.com/paper/2510.13161/) (2025)
- [TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference](https://mlsanity.com/paper/2602.05145/) (2026)
- [Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding](https://mlsanity.com/paper/2603.05210/) (2026)
- [Acceptance Dynamics Across Cognitive Domains in Speculative Decoding](https://mlsanity.com/paper/2604.14682/) (2026)
- [EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation](https://mlsanity.com/paper/2605.27390/) (2026)
- [Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding](https://mlsanity.com/paper/2609.09338/) (2026)
- [Recovering Off-Policy Supervision for Speculative Decoding](https://mlsanity.com/paper/2609.38795/) (2026)
- [LEAP: Learning Efficient Action Proposals For LLM Agents](https://mlsanity.com/paper/2610.02670/) (2026)

## Abstract (arXiv)

Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. However, when adapting speculative decoding to domain-specific target models, the acceptance rate of the generic draft model drops significantly due to domain shift. In this work, we systematically investigate knowledge distillation techniques for training domain draft models to improve their speculation accuracy. We compare white-box and black-box distillation approaches and explore their effectiveness in various data accessibility scenarios, including historical user queries, curated domain data, and synthetically generated alignment data. Our experiments across Function Calling, Biology, and Chinese domains show that offline distillation consistently outperforms online distillation by 11% to 25%, white-box distillation surpasses black-box distillation by 2% to 10%, and data scaling trends hold across domains. Additionally, we find that synthetic data can effectively align draft models and achieve 80% to 93% of the performance of training on historical user queries. These findings provide practical guidelines for training domain-specific draft models to improve speculative decoding efficiency.