mlsanityData to 7 Oct 2026

Freeform Preference Learning for Robotic Manipulation

arXiv 2606.32027 ↗First posted 30 Jun 2026cs.RO cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 51 of its 58 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

51 of 58 references.

References · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
38 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  2. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  10. 2024Diffusion Policy Policy Optimizationcited by 255
  11. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  12. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  13. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  14. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  15. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  16. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  17. 2019Reward-Conditioned Policiescited by 101
  18. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  19. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  20. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  21. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  22. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  23. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  24. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  25. 2026RoboReward: General-Purpose Vision-Language Reward Models for Roboticscited by 46
  26. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  27. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  28. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  29. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  30. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  31. 2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
  32. 2022Inducing Structure in Reward Learning by Learning Featurescited by 16
  33. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  34. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  35. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
  36. 2024Preference-Conditioned Language-Guided Abstractioncited by 11
  37. 2024Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Inputcited by 8
  38. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Logic-VLA: A Temporal Logic Conditioned Vision-Language-Action Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
  2. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning
  3. 2024Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
  4. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
  5. 2025FDPP: Fine-tune Diffusion Policy with Human Preference
  6. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preference
  7. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
  8. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models