mlsanityData to 14 Oct 2026

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models

arXiv 2505.19223 ↗First posted 25 May 2025cs.LGcited by 285

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

While Masked Diffusion Models (MDMs), such as LLaDA, present a promising paradigm for language modeling, there has been relatively little effort in aligning these models with human preferences via reinforcement learning. The challenge primarily arises from the high variance in Evidence Lower Bound (ELBO)-based likelihood estimates required for preference optimization. To address this issue, we propose Variance-Reduced Preference Optimization (VRPO), a framework that formally analyzes the variance of ELBO estimators and derives bounds on both the bias and variance of preference optimization gradients. Building on this theoretical foundation, we introduce unbiased variance reduction strategies, including optimal Monte Carlo budget allocation and antithetic sampling, that significantly improve the performance of MDM alignment. We demonstrate the effectiveness of VRPO by applying it to LLaDA, and the resulting model, LLaDA 1.5, outperforms its SFT-only predecessor consistently and significantly across mathematical (GSM8K +4.7), code (HumanEval +3.0, MBPP +1.8), and alignment benchmarks (IFEval +4.0, Arena-Hard +4.3). Furthermore, LLaDA 1.5 demonstrates a highly competitive mathematical performance compared to strong language MDMs and ARMs. Project page: https://ml-gsai.github.io/LLaDA-1.5-Demo/.

Key facts. Cited by 285 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 64 of its 85 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

64 of 85 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Preference-Based Alignment of Discrete Diffusion Models11 Mar
  2. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods3 Feb
  3. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  4. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  6. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  7. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,963
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  13. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  14. 2020GLU Variants Improve Transformercited by 1,804
  15. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  16. 2019Root Mean Square Layer Normalizationcited by 1,480
  17. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  20. 2025Large Language Diffusion Modelscited by 912
  21. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  22. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  23. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  24. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  25. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  26. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  27. 2015Importance Weighted Autoencoderscited by 456
  28. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  29. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  30. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  31. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  32. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  33. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  34. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  35. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  36. 2024Scaling up Masked Diffusion Models on Textcited by 199
  37. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  38. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 148

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 98; 2026: 187.

285 papers.

Sort:

By area:

  1. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  5. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 116
  6. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  7. 2025A Survey on Latent Reasoningcited by 81
  8. 2025A Survey on Diffusion Language Modelscited by 78
  9. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  10. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  11. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  12. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
38 more
  1. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  2. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  3. 2025Diffusion Language Models are Super Data Learnerscited by 48
  4. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 43
  5. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  6. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  7. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  8. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  9. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 34
  10. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 34
  11. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  12. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  13. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 33
  14. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  15. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  16. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  17. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  18. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  19. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 30
  20. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  21. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 26
  22. 2025Training Optimal Large Diffusion Language Modelscited by 25
  23. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  24. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  25. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 23
  26. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 22
  27. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  28. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  29. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Modelscited by 19
  30. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 19
  31. 2025OBS-Diff: Accurate Pruning For Diffusion Models in One-Shotcited by 19
  32. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
  33. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  34. 2026The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Modelscited by 18
  35. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  36. 2025Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Stepcited by 17
  37. 2025SparseD: Sparse Attention for Diffusion Language Modelscited by 17
  38. 2025Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Modelscited by 17

50 most cited of 285 All 285 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  2. 2025On the Role of Preference Variance in Preference Optimization
  3. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  4. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  5. 2024Multi-Reference Preference Optimization for Large Language Models
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward
  8. 2025A Survey of Direct Preference Optimization