mlsanityData to 14 Oct 2026

Multi-Token Residual Prediction

arXiv 2605.18817 ↗First posted 12 May 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion Language Models (DLMs) generate text by iteratively denoising masked token sequences, offering a tradeoff between parallelism and quality compared to autoregressive models. In current practice, the number of tokens decoded per step is controlled by a confidence threshold, and quality degrades monotonically as more tokens are denoised per step. We introduce Multi-token Residual Prediction (MRP), a lightweight module that enables dependency-aware multi-token denoising within a single backbone forward pass. MRP exploits a key property of the denoising process: the logit distributions at adjacent denoising steps are remarkably similar. Rather than running the backbone a second time to obtain the next-step logits, MRP predicts the residual between steps from the backbone's hidden states, effectively denoising more tokens per backbone forward at a fraction of the cost. We apply MRP across the two operating regimes of DLM decoding. In the high-quality-low-throughput static denoising regime, MRP serves as a drafter for speculative decoding: its proposals are verified against the backbone, yielding lossless acceleration of up to 1.4x in SGLang. In the low-quality-high-throughput dynamic denoising regime, MRP instead drives a remasking scheme that revokes over-eager reveals, recovering most of the accuracy lost to aggressive low-threshold decoding and improving accuracy by up to 22.6 points on code generation task HumanEval and 17.7 points on reasoning task GSM8K.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

35 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Residual Context Diffusion Language Models30 Jan
  2. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules2 Dec
  3. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
  4. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding28 Sep
  5. 2025Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning18 Sep
  6. 2025Diffusion Language Models Know the Answer Before Decoding27 Aug
  7. 2025Dream 7B: Diffusion Large Language Models21 Aug
  8. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  10. 2025Large Language Diffusion Models14 Feb
  11. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
11 more
  1. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  2. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  3. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  4. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  5. 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 50
  6. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 46
  7. 2024GenQA: Generating Millions of Instructions from a Handful of Promptscited by 35
  8. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 30
  9. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 27
  10. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 22
  11. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Trajectory-Level Speculative Decoding for Diffusion Language Models
  2. 2026Know When to Hold 'em: Correct-Token Retention in Uniform-State Diffusion Language Models
  3. 2026SpecFold: Folding Multi-Branch Redundancy for Faster Speculative Decoding in Diffusion Language Models
  4. 2026Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
  5. 2026Supportive Token Revealing for Fast Diffusion Language Model Decoding
  6. 2026PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
  7. 2026Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast
  8. 2026Stability-Weighted Decoding for Diffusion Language Models