mlsanityData to 14 Oct 2026

OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding

arXiv 2507.02659 ↗First posted 3 Jul 2025cs.LG cs.CLcited by 4

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding generally dictates having a small, efficient draft model that is either pretrained or distilled offline to a particular target model series, for instance, Llama or Qwen models. However, within online deployment settings, there are two major challenges: 1) usage of a target model that is incompatible with the draft model; 2) expectation of latency improvements over usage and time. In this work, we propose OmniDraft, a unified framework that enables a single draft model to operate with any target model and adapt dynamically to user data. We introduce an online n-gram cache with hybrid distillation fine-tuning to address the cross-vocabulary mismatch across draft and target models; and further improve decoding speed by leveraging adaptive drafting techniques. OmniDraft is particularly suitable for on-device LLM applications where model cost, efficiency and user customization are the major points of contention. This further highlights the need to tackle the above challenges and motivates the textit``one drafter for all'' paradigm. We showcase the proficiency of the OmniDraft framework by performing online learning on math reasoning, coding and text generation tasks. Notably, OmniDraft enables a single Llama-68M model to pair with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding; and additionally provides up to 1.5-2x speedup.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 52 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

47 of 52 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025RASD: Retrieval-Augmented Speculative Decoding5 Mar
  2. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies31 Jan
  3. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability24 Oct
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  5. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  6. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  8. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  9. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models7 May
  10. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  11. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  12. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
8 more
  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  5. 2023Online Speculative Decoding11 Oct
  6. 2023Speculative Decoding with Big Little Decoder15 Feb
  7. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
15 more
  1. 2016Sequence-Level Knowledge Distillationcited by 801
  2. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  3. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  4. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  5. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  6. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  7. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  8. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  9. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  10. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  11. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  12. 2024Zero-Shot Tokenizer Transfercited by 31
  13. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  14. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  15. 2024Online Training of Large Language Models: Learn while chattingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  2. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  3. 2026X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decodingcited by 0
  4. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
  2. 2026MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
  3. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  4. 2024Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
  5. 2024Decoding Speculative Decoding
  6. 2025DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
  7. 2026AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices
  8. 2025Mamba Drafters for Speculative Decoding