mlsanityData to 7 Oct 2026

Multi-Token Prediction via Self-Distillation

arXiv 2602.06019 ↗First posted 5 Feb 2026cs.CL cs.LGcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Existing techniques for accelerating language model inference, such as speculative decoding, require training auxiliary speculator models and building and deploying complex inference pipelines. We consider a new approach for converting a pretrained autoregressive language model from a slow single next token prediction model into a fast standalone multi-token prediction model using a simple online distillation objective. The final model retains the exact same implementation as the pretrained initial checkpoint and is deployable without the addition of any auxiliary verifier or other specialized inference code. Our method produces models that decode more than 3× faster at <5% drop in accuracy on GSM8K relative to the single token decoding performance of the same checkpoint.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 28 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

23 of 28 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Speculative Decoding: Performance or Illusion?31 Dec
  2. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction16 Sep
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  9. 2025Deep Think with Confidencecited by 199
  10. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  11. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  12. 2024The pitfalls of next-token predictioncited by 156
4 more
  1. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  2. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 14
  3. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  4. 2025Parallel Token Prediction for Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 2
  2. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Draftingcited by 1
  3. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  4. 2026Dynamic Multi-Byte Prediction With Hierarchical Language Modelscited by 0
  5. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  6. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decodercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Distillation for Multi-Token Prediction
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2025Set Block Decoding is a Language Model Inference Accelerator
  4. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  5. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  6. 2025Pre-Training Curriculum for Multi-Token Prediction in Language Models
  7. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  8. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding