mlsanityData to 7 Oct 2026

FDPP: Fine-tune Diffusion Policy with Human Preference

arXiv 2501.08259 ↗First posted 14 Jan 2025cs.RO cs.LGcited by 14

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Imitation learning from human demonstrations enables robots to perform complex manipulation tasks and has recently witnessed huge success. However, these techniques often struggle to adapt behavior to new preferences or changes in the environment. To address these limitations, we propose Fine-tuning Diffusion Policy with Human Preference (FDPP). FDPP learns a reward function through preference-based learning. This reward is then used to fine-tune the pre-trained policy with reinforcement learning (RL), resulting in alignment of pre-trained policy with new human preferences while still solving the original task. Our experiments across various robotic tasks and preferences demonstrate that FDPP effectively customizes policy behavior without compromising performance. Additionally, we show that incorporating Kullback-Leibler (KL) regularization during fine-tuning prevents over-fitting and helps maintain the competencies of the initial policy.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 30 of its 39 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

30 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Diffusion Policy Policy Optimization1 Sep
  2. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations6 Mar
  3. 2023Training Diffusion Models with Reinforcement Learning22 May
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  12. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
14 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  5. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  6. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  7. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  8. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  9. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  10. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  11. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  12. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  13. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Rankingcited by 8
  14. 2024Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Gamescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  2. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  3. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  4. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 14
  5. 2025EvolvingGrasp: Evolutionary Grasp Generation via Efficient Preference Alignmentcited by 6
  6. 2025D2PPO: Diffusion Policy Policy Optimization with Dispersive Losscited by 5
  7. 2026PPGuide: Steering Diffusion Policies with Performance Predictive Guidancecited by 2
  8. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 1
  9. 2026How to Peel with a Knife: Aligning Fine-Grained Manipulation with Human Preferencecited by 1
  10. 2026Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Correctionscited by 1
  11. 2026Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steeringcited by 1
  12. 2026Logic-VLA: A Temporal Logic Conditioned Vision-Language-Action Modelcited by 0
2 more
  1. 2026Latent evolving World Action Modelcited by 0
  2. 2026PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Freeform Preference Learning for Robotic Manipulation
  2. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preference
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning
  4. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning
  5. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning
  6. 2024Active Fine-Tuning of Multi-Task Policies
  7. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
  8. 2024Unpacking the Individual Components of Diffusion Policy