# Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2602.09583 · HTML: https://mlsanity.com/paper/2602.09583/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Aligning Diffusion Models by Optimizing Human Utility](https://mlsanity.com/paper/2404.04465/) (2024)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [Imitating Task and Motion Planning with Visuomotor Transformers](https://mlsanity.com/paper/2305.16309/) (2023)
- [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019)
- [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021)
- [Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts](https://mlsanity.com/paper/2402.10958/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization](https://mlsanity.com/paper/2406.06382/) (2024)
- [Learning Preferences for Manipulation Tasks from Online Coactive Feedback](https://mlsanity.com/paper/1601.00741/) (2016)
- [The Crucial Role of Samplers in Online Direct Preference Optimization](https://mlsanity.com/paper/2409.19605/) (2024)
- [Robot Learning as an Empirical Science: Best Practices for Policy Evaluation](https://mlsanity.com/paper/2409.09491/) (2024)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Unfolding the Literature: A Review of Robotic Cloth Manipulation](https://mlsanity.com/paper/2407.01361/) (2024)
- [A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Models](https://mlsanity.com/paper/2403.16730/) (2024)
- [A Virtual Reality Framework for Human-Robot Collaboration in Cloth Folding](https://mlsanity.com/paper/2305.07493/) (2023)
- [DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment](https://mlsanity.com/paper/2410.11584/) (2024)
- [Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control](https://mlsanity.com/paper/2410.13586/) (2024)
- [Real-Time Operator Takeover for Visuomotor Diffusion Policy Training](https://mlsanity.com/paper/2502.02308/) (2025)
- [Visual Preference Inference: An Image Sequence-Based Preference Reasoning in Tabletop Object Manipulation](https://mlsanity.com/paper/2403.11513/) (2024)

## Built on by (most cited first, 2 of 2)

- [SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)](https://mlsanity.com/paper/2607.04234/) (2026)
- [Dressing in Motion: A Human Motion-Aware Diffusion Policy for Robot-Assisted Dressing](https://mlsanity.com/paper/2609.04759/) (2026)

## Abstract (arXiv)

Humans naturally develop preferences for how manipulation tasks should be performed, which are often subtle, personal, and difficult to articulate. Although it is important for robots to account for these preferences to increase personalization and user satisfaction, they remain largely underexplored in robotic manipulation, particularly in the context of deformable objects like garments and fabrics. In this work, we study how to adapt pretrained visuomotor diffusion policies to reflect preferred behaviors using limited demonstrations. We introduce RKO, a novel preference-alignment method that combines the benefits of two recent frameworks: RPO and KTO. We evaluate RKO against common preference learning frameworks, including these two, as well as a baseline vanilla diffusion policy, on real-world cloth-folding tasks spanning multiple garments and preference settings. We show that preference-aligned policies (particularly RKO) achieve superior performance and sample efficiency compared to standard diffusion policy fine-tuning. These results highlight the importance and feasibility of structured preference learning for scaling personalized robot behavior in complex deformable object manipulation tasks.