# Drifting Objectives for Refining Discrete Diffusion Language Models

arXiv: https://arxiv.org/abs/2605.19470 · HTML: https://mlsanity.com/paper/2605.19470/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Discrete and text diffusion, one of 14 directions in Diffusion models.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 19 May 2026.

## Placed in

- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 27 of 27)

- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Dream 7B: Diffusion Large Language Models](https://mlsanity.com/paper/2508.15487/) (2025)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching](https://mlsanity.com/paper/2506.06295/) (2025)
- [dKV-Cache: The Cache for Diffusion Language Models](https://mlsanity.com/paper/2505.15781/) (2025)
- [The Diffusion Duality](https://mlsanity.com/paper/2506.10892/) (2025)
- [Generative Modeling via Drifting](https://mlsanity.com/paper/2602.04770/) (2026)
- [Dirichlet Flow Matching with Applications to DNA Sequence Design](https://mlsanity.com/paper/2402.05841/) (2024)
- [Accelerating Diffusion LLMs via Adaptive Parallel Decoding](https://mlsanity.com/paper/2506.00413/) (2025)
- [Beyond Autoregression: Fast LLMs via Self-Distillation Through Time](https://mlsanity.com/paper/2410.21035/) (2024)
- [Flow Map Language Models: One-step Language Modeling via Continuous Denoising](https://mlsanity.com/paper/2602.16813/) (2026)
- [Distillation of Discrete Diffusion through Dimensional Correlations](https://mlsanity.com/paper/2410.08709/) (2024)
- [Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles](https://mlsanity.com/paper/2506.10848/) (2025)
- [Categorical Flow Maps](https://mlsanity.com/paper/2602.12233/) (2026)
- [Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models](https://mlsanity.com/paper/2506.19037/) (2025)
- [Discrete Flow Maps](https://mlsanity.com/paper/2604.09784/) (2026)
- [Scaling Beyond Masked Diffusion Language Models](https://mlsanity.com/paper/2602.15014/) (2026)
- [FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models](https://mlsanity.com/paper/2509.20624/) (2025)
- [Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct](https://mlsanity.com/paper/2509.25035/) (2025)
- [The Diffusion Duality, Chapter II: Psi-Samplers](https://mlsanity.com/paper/2602.21185/) (2026)
- [Stopping Computation for Converged Tokens in Masked Diffusion-LM Decoding](https://mlsanity.com/paper/2602.06412/) (2026)

## Built on by (most cited first, 2 of 2)

- [One-Step Generation via Riemannian Wasserstein Gradient Flows](https://mlsanity.com/paper/2610.04454/) (2026)
- [Representation-Space MMD for Diffusion Language Models](https://mlsanity.com/paper/2610.06648/) (2026)

## Abstract (arXiv)

Discrete diffusion language models (DDLMs) generate text by iteratively denoising categorical token sequences, while recent drifting methods for continuous generators suggest that part of this sampling-time correction can instead be absorbed into training through an anti-symmetric fixed-point objective. We study how to transfer this principle to DDLMs, where the main challenge is the interface with discrete text: hard token samples are non-differentiable, and categorical predictions do not directly provide continuous samples to drift. We formulate TokenDrift, a drifting objective that lifts categorical predictions to soft-token features, applies anti-symmetric drifting in a frozen semantic space, and backpropagates the resulting stop-gradient feature target to DDLM logits. In controlled continual-training experiments with masked and uniform-state diffusion backbones, TokenDrift improves fixed-NFE generation quality over matched continuation baselines, reducing Gen.-PPL at 4 NFEs by 89% on MDLM and 86% on DUO. These results suggest that drifting can provide a practical refinement objective for DDLMs.