# Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

arXiv: https://arxiv.org/abs/2406.16758 · HTML: https://mlsanity.com/paper/2406.16758/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding](https://mlsanity.com/paper/2310.05424/) (2023)
- [Speculative Streaming: Fast LLM Inference without Auxiliary Models](https://mlsanity.com/paper/2402.11131/) (2024)
- [Multi-Candidate Speculative Decoding](https://mlsanity.com/paper/2401.06706/) (2024)
- [JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus](https://mlsanity.com/paper/2202.12607/) (2022)
- [Discovering Representation Sprachbund For Multilingual Pre-Training](https://mlsanity.com/paper/2109.00271/) (2021)

## Built on by (most cited first, 12 of 12)

- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [Mixture of Attentions For Speculative Decoding](https://mlsanity.com/paper/2410.03804/) (2024)
- [Guiding Reasoning in Small Language Models with LLM Assistance](https://mlsanity.com/paper/2504.09923/) (2025)
- [Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs](https://mlsanity.com/paper/2510.20064/) (2025)
- [Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding](https://mlsanity.com/paper/2502.05609/) (2025)
- [Make Every Draft Count: Hidden State based Speculative Decoding](https://mlsanity.com/paper/2602.21224/) (2026)
- [Multi-Drafter Speculative Decoding with Alignment Feedback](https://mlsanity.com/paper/2604.05417/) (2026)
- [SSSD: Simply-Scalable Speculative Decoding](https://mlsanity.com/paper/2411.05894/) (2024)
- [The Disparate Impacts of Speculative Decoding](https://mlsanity.com/paper/2510.02128/) (2025)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [Speculative Decoding and the Curse of Multilinguality](https://mlsanity.com/paper/2605.30580/) (2026)
- [MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices](https://mlsanity.com/paper/2608.10362/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.