mlsanityData to 7 Oct 2026

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

arXiv 2406.16758 ↗First posted 24 Jun 2024cs.CLcited by 12

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 43 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

27 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  6. 2024Multi-Candidate Speculative Decoding12 Jan
  7. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  8. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  10. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Sequence-Level Knowledge Distillationcited by 761
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
4 more
  1. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  3. 2022JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpuscited by 17
  4. 2021Discovering Representation Sprachbund For Multilingual Pre-Trainingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 5.

12 papers.

  1. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  2. 2024Mixture of Attentions For Speculative Decodingcited by 17
  3. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 13
  4. 2025Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMscited by 5
  5. 2025Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decodingcited by 4
  6. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  7. 2026Multi-Drafter Speculative Decoding with Alignment Feedbackcited by 3
  8. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  9. 2025The Disparate Impacts of Speculative Decodingcited by 2
  10. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  11. 2026Speculative Decoding and the Curse of Multilingualitycited by 0
  12. 2026MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  3. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  4. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2023Accelerating LLM Inference with Staged Speculative Decoding
  7. 2026Faster LLM Inference via Sequential Monte Carlo
  8. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference