mlsanityData to 14 Oct 2026

Aligning Diffusion Language Models via Unpaired Preference Optimization

arXiv 2510.23658 ↗First posted 26 Oct 2025cs.LG cs.AIcited by 1

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion language models (dLLMs) are an emerging alternative to autoregressive (AR) generators, but aligning them to human preferences is challenging because sequence log-likelihoods are intractable and pairwise preference data are costly to collect. We introduce ELBO-KTO, which combines an ELBO surrogate for diffusion log-likelihoods with a prospect-theoretic, unpaired preference objective (Kahneman Tversky Optimization, KTO). We analyze the bias and variance induced by the ELBO substitution and employ variance-reduction practices that stabilize gradients during training. Applied to LLaDA-8B-Instruct, ELBO-KTO yields 65.9% and 62.3% adjusted win rates on kto-mix-14k and UltraFeedback-Binary, respectively, versus the base model under an automatic LLM judge. Across downstream tasks, including GSM8K, MMLU, and additional reasoning/knowledge benchmarks, ELBO-KTO trained on UltraFeedback-Binary performs on par with or better than the base model under identical decoding. This establishes unpaired preference optimization as a viable alternative to pairwise alignment in diffusion LLMs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 25 of its 28 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

25 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models25 May
  2. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Models4 Feb
  3. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Reference10 Jun
  4. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  5. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2025Large Language Diffusion Modelscited by 912
  10. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  11. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  12. 2025Dream 7B: Diffusion Large Language Modelscited by 544
8 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  4. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  5. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  6. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  7. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  8. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Backdooring Masked Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
  2. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
  3. 2025Rethinking Direct Preference Optimization in Diffusion Models
  4. 2023Likelihood-Based Diffusion Language Models
  5. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problem
  6. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  7. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignment
  8. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributions