mlsanityData to 14 Oct 2026

SelFusion: Self-distillation for Diffusion Language Models

arXiv 2608.22898 ↗First posted 24 Aug 2026cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion language models (DLMs) alleviate the inherent latency bottleneck of autoregressive (AR) large language models (LLMs), but their degraded generation quality limits practical applicability. Although knowledge distillation (KD) can be a promising direction for improving performance, we empirically find that naively applying conventional KD yields only marginal gains, or even degrades generation quality. Based on these observations, we propose a novel self-distillation framework for DLMs, namely SelFusion. To enable effective KD without an external teacher model, SelFusion performs two forward passes with different masking levels, defining the hard mode with a larger masking probability and the easy mode with a smaller masking probability. However, the easy mode is not always more accurate than the hard mode and can be overconfident on incorrect tokens. Thus, we introduce bidirectional KD between the two modes, which can dynamically determine the distillation direction based on token-level correctness. Experimental results on instruction-following tasks show that the proposed self-distillation substantially outperforms other KD methods with external LLM and DLM teachers. In many configurations, the student trained with SelFusion even surpasses the performance of the LLM teacher, providing a practical path toward improving DLM generation quality. Source code can be found at https://github.com/scai-research/SelFusion_official

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 15 of its 24 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

15 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generation30 May
  2. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  3. 2024Scaling up Masked Diffusion Models on Text24 Oct
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 11

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  8. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  9. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  10. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 25
  11. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasonerscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models
  2. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  3. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
  4. 2026Learning from the Self-future: On-policy Self-distillation for dLLMs
  5. 2026Adaptation to Intrinsic Dependence in Diffusion Language Models
  6. 2026Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models
  7. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
  8. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation