mlsanityData to 7 Oct 2026

Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

arXiv 2502.01384 ↗First posted 3 Feb 2025stat.ML cs.AI cs.CLcited by 36

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Discrete diffusion models have recently gained significant attention due to their ability to process complex discrete structures for language modeling. However, fine-tuning these models with policy gradient methods, as is commonly done in Reinforcement Learning from Human Feedback (RLHF), remains a challenging task. We propose an efficient, broadly applicable, and theoretically justified policy gradient algorithm, called Score Entropy Policy Optimization (SEPO), for fine-tuning discrete diffusion models over non-differentiable rewards. Our numerical experiments across several discrete generative tasks demonstrate the scalability and efficiency of our method. Our code is available at https://github.com/ozekri/SEPO.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 46 of its 74 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

46 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Diffusion Policy Policy Optimization1 Sep
  2. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review18 Jul
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  4. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  5. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
29 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  8. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  9. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  10. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  11. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  12. 2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
  13. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  14. 2021Efficient and Modular Implicit Differentiationcited by 199
  15. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  16. 2023Protein Design with Guided Discrete Diffusioncited by 154
  17. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  18. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  19. 2024Dirichlet Flow Matching with Applications to DNA Sequence Designcited by 110
  20. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  21. 2022From Denoising Diffusions to Denoising Markov Modelscited by 58
  22. 2024Informed Correctors for Discrete Diffusion Modelscited by 57
  23. 2024Large Language Models as Markov Chainscited by 33
  24. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  25. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  26. 2024Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysiscited by 24
  27. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 22
  28. 2024Mirror and Preconditioned Gradient Descent in Wasserstein Spacecited by 11
  29. 2024Learning with Fitzpatrick Lossescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 17.

36 papers.

Sort:

By area:

  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  3. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  4. 2025A Survey on Diffusion Language Modelscited by 75
  5. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  6. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  7. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 30
  8. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  9. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 21
  10. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 17
  11. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 15
  12. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
24 more
  1. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  2. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  3. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  4. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 10
  5. 2026Discrete Adjoint Matchingcited by 10
  6. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spacescited by 7
  7. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  8. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  9. 2026Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Modelscited by 3
  10. 2026Diffusion-State Policy Optimization for Masked Diffusion Language Modelscited by 3
  11. 2026Reward-Guided Discrete Diffusion via Clean-Sample Markov Chain for Molecule and Biological Sequence Designcited by 3
  12. 2026Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusioncited by 3
  13. 2026Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correctioncited by 3
  14. 2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 2
  15. 2026Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judgecited by 2
  16. 2026NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMscited by 2
  17. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Designcited by 2
  18. 2026A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decodingcited by 2
  19. 2026Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learningcited by 1
  20. 2026Temporal Difference Learning for Diffusion Modelscited by 1
  21. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  22. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0
  23. 2026Discrete Flow Matching Policy Optimizationcited by 0
  24. 2026Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning
  2. 2026Entropy-Preserving Reinforcement Learning
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
  5. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  6. 2023Score Regularized Policy Optimization through Diffusion Behavior
  7. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence
  8. 2026Diffusion Policy through Conditional Proximal Policy Optimization