mlsanityData to 14 Oct 2026

AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

arXiv 2310.02054 ↗First posted 3 Oct 2023cs.AIcited by 37

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RL from Human Feedback (RLHF) to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on https://aligndiff.github.io/.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 48 of its 60 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

48 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL31 May
  2. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Models30 May
  3. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  4. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 42

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
30 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  12. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 337
  13. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  14. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  15. 2017Inverse Reward Designcited by 249
  16. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  17. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  18. 2018Zero-Shot Visual Imitationcited by 117
  19. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  20. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  21. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  22. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  23. 2021What Matters for Adversarial Imitation Learning?cited by 70
  24. 2023MolDiff: Addressing the Atom-Bond Inconsistency Problem in 3D Molecule Diffusion Generationcited by 65
  25. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  26. 2021Hierarchical Skills for Efficient Explorationcited by 38
  27. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  28. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  29. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  30. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferencescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 18; 2025: 16; 2026: 2.

37 papers.

Sort:

By area:

  1. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  2. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  3. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  4. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  5. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 42
  6. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  7. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  8. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 38
  9. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  10. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 36
  11. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  12. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
25 more
  1. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
  2. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  3. 2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
  4. 2025DDAT: Diffusion Policies Enforcing Dynamically Admissible Robot Trajectoriescited by 14
  5. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  6. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  7. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  8. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  9. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  10. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  11. 2025BalancedDPO: Adaptive Multi-Metric Alignmentcited by 6
  12. 2025Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Alignercited by 5
  13. 2024MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learningcited by 4
  14. 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 3
  15. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  16. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learningcited by 2
  17. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  18. 2025FlexiFlow: decomposable flow matching for generation of flexible molecular ensemblecited by 2
  19. 2026Compositional Visual Planning via Inference-Time Diffusion Scalingcited by 2
  20. 2025DnD Filter: Differentiable State Estimation for Dynamic Systems using Diffusion Modelscited by 1
  21. 2025FRMD: Fast Robot Motion Diffusion with Consistency-Distilled Movement Primitives for Smooth Action Generationcited by 1
  22. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1
  23. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
  24. 2025What Do You Need for Compositional Generalization in Diffusion Planning?cited by 0
  25. 2025Fast and Stable Diffusion Planning through Variational Adaptive Weightingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Embedding Adaptation for Human Preference Alignment in Diffusion Planners
  2. 2025Influencing Humans to Conform to Preference Models for RLHF
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  4. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
  5. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  6. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  7. 2024PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers
  8. 2026A Markov Chain Approach to Preference Alignment