mlsanityData to 14 Oct 2026

iADD: Improving Alignment and Diversity in Diffusion Policy Optimization

arXiv 2610.01789 ↗First posted 1 Oct 2026cs.LG cs.AIcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Reinforcement learning based post training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimizations do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design. We analyze the theoretical framework and mathematically demonstrate that only-latter timestep updates of diffusion model may be harmful for diversity contrary to the conclusions presented in a previous work. Additionally, we propose an incremental Feynman-Kac training based on strong theoretical foundations in order to achieve the best-yet alignment-diversity tradeoffs. We perform extensive experiments and compare our method against related diffusion policy optimization approaches in three different tasks and also provide strong ablations for each component, thus validating strong performance gains in both alignment and diversity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 57 of its 71 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

57 of 71 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models7 Sep
  2. 2025Rethinking Direct Preference Optimization in Diffusion Models24 May
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Steerable Scene Generation with Post Training and Inference-Time Search7 May
  5. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards14 Mar
  6. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning10 Oct
  7. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models9 Sep
  8. 2024Aligning Diffusion Models by Optimizing Human Utility6 Apr
  9. 2024Large-scale Reinforcement Learning for Diffusion Models20 Jan
  10. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  11. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  12. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
3 more
  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  2. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  3. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
30 more
  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  12. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  13. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  14. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  15. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  16. 2022Perception Prioritized Training of Diffusion Modelscited by 303
  17. 2022Convergence of denoising diffusion models under the manifold hypothesiscited by 242
  18. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  19. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  20. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  21. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  22. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 76
  23. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
  24. 2019Dealing with Sparse Rewards in Reinforcement Learningcited by 42
  25. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
  26. 2024Mixed Diffusion for 3D Indoor Scene Synthesiscited by 32
  27. 2025Convex Relaxation for Robust Vanishing Point Estimation in Manhattan Worldcited by 15
  28. 2024Harmonic Path Integral Diffusioncited by 11
  29. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
  30. 2025Inferring Compositional 4D Scenes without Ever Seeing Onecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dichotomous Diffusion Policy Optimization
  2. 2026DiPOD: Diffusion Policy Optimization without Drifting Apart
  3. 2026Explicit Critic Guidance for Aligning Diffusion Models
  4. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
  5. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  6. 2026Step-level Denoising-time Diffusion Alignment with Multiple Objectives
  7. 2026Diffusion Controller: Framework, Algorithms and Parameterization
  8. 2026Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser