mlsanityData to 14 Oct 2026

Latent Embedding Adaptation for Human Preference Alignment in Diffusion Planners

arXiv 2503.18347 ↗First posted 24 Mar 2025cs.LG cs.AI cs.ROcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This work addresses the challenge of personalizing trajectories generated in automated decision-making systems by introducing a resource-efficient approach that enables rapid adaptation to individual users' preferences. Our method leverages a pretrained conditional diffusion model with Preference Latent Embeddings (PLE), trained on a large, reward-free offline dataset. The PLE serves as a compact representation for capturing specific user preferences. By adapting the pretrained model using our proposed preference inversion method, which directly optimizes the learnable PLE, we achieve superior alignment with human preferences compared to existing solutions like Reinforcement Learning from Human Feedback (RLHF) and Low-Rank Adaptation (LoRA). To better reflect practical applications, we create a benchmark experiment using real human preferences on diverse, high-reward trajectories.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 73 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

54 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2022Models of human preference for learning reward functions5 Jun
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning27 May
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
34 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2022A Generalist Agentcited by 808
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  13. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  14. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  15. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  16. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  17. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  18. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  19. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  20. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  21. 2021GAN Inversion: A Surveycited by 367
  22. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  23. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  24. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  25. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  26. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  27. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  28. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 98
  29. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  30. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  31. 2020Self-Driving like a Human driver instead of a Robocar: Personalized comfortable driving experience for autonomous vehiclescited by 20
  32. 2018Personalized Dynamics Models for Adaptive Assistive Navigation Systemscited by 13
  33. 2023Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environmentscited by 4
  34. 2020A Data-Driven Reinforcement Learning Solution Framework for Optimal and Adaptive Personalization of a Hip Exoskeletoncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedbackcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
  2. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
  3. 2026PAPA: Online Personalized Active Preference Alignment
  4. 2026In-Context Reward Adaptation for Robust Preference Modeling
  5. 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
  6. 2024LRHP: Learning Representations for Human Preferences via Preference Pairs
  7. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences