mlsanityData to 7 Oct 2026

Few-Step Diffusion Language Models via Trajectory Self-Distillation

arXiv 2602.12262 ↗First posted 12 Feb 2026cs.CL cs.LGcited by 3

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding. However, realizing this potential in practice remains challenging: reducing the number of decoding steps, typically causes a substantial degradation in output quality due to token factorization error. To alleviate this, we propose a self-distillation framework that trains a few-step student to match the generative trajectory of a full-step teacher. We theoretically and empirically show that trajectory-level supervision mitigates this factorization error, thereby enabling effective few-step decoding. We further incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that encourages mode-seeking toward the teacher's modes, yielding stronger performance on challenging reasoning tasks. Across reasoning and code-generation benchmarks, our method substantially narrows the gap between few-step and full-step decoding. The source code is available at https://github.com/Tyrion58/T3D.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

45 of 53 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025CDLM: Consistency Diffusion Language Models For Faster Sampling24 Nov
  2. 2025WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning22 Nov
  3. 2025Variational Masked Diffusion Models27 Oct
  4. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
  5. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models8 Sep
  6. 2025Dream 7B: Diffusion Large Language Models21 Aug
  7. 2025A Survey on Diffusion Language Models14 Aug
  8. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  9. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  10. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion27 May
  11. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  12. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
4 more
  1. 2025Large Language Diffusion Models14 Feb
  2. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  3. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun

Other references · 29

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Consistency Modelscited by 1,778
  10. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  11. 2025Mean Flows for One-step Generative Modelingcited by 524
  12. 2023Improved Techniques for Training Consistency Modelscited by 430
17 more
  1. 2015Policy Distillationcited by 414
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  5. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  6. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  7. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  9. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  10. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  11. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
  12. 2024INTELLECT-1 Technical Reportcited by 20
  13. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  14. 2025CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Modelscited by 18
  15. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  16. 2025ReDi: Rectified Discrete Flowcited by 13
  17. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Modelscited by 1
  2. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 0
  3. 2026Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TRIMS: Trajectory-Ranked Instruction Masked Supervision for Diffusion Language Models
  2. 2026OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models
  3. 2026Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models
  4. 2026Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models
  5. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
  6. 2026Improving Diffusion Language Model Decoding through Joint Search in Generation Order and Token Space
  7. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  8. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generation