mlsanityData to 7 Oct 2026

GRAPE: Generalizing Robot Policy via Preference Alignment

arXiv 2411.19309 ↗First posted 28 Nov 2024cs.RO cs.CV cs.LGcited by 106

On the map: Vision-language-action models → Reinforcement learning and post-training

also in 1 more
Abstract (arXiv)

Despite the recent advancements of vision-language-action (VLA) models on a variety of robotics tasks, they suffer from critical issues such as poor generalizability to unseen tasks, due to their reliance on behavior cloning exclusively from successful rollouts. Furthermore, they are typically fine-tuned to replicate demonstrations collected by experts under different settings, thus introducing distribution bias and limiting their adaptability to diverse manipulation objectives, such as efficiency, safety, and task completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level and implicitly models reward from both successful and failure trials to boost generalizability to diverse tasks. Moreover, GRAPE breaks down complex manipulation tasks to independent stages and automatically guides preference modeling through customized spatiotemporal constraints with keypoints proposed by a large vision-language model. Notably, these constraints are flexible and can be customized to align the model with varying objectives, such as safety, efficiency, or task success. We evaluate GRAPE across a diverse array of tasks in both real-world and simulated environments. Experimental results demonstrate that GRAPE enhances the performance of state-of-the-art VLA models, increasing success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively. Additionally, GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively. All code, models, and data are available at https://grape-vla.github.io/

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 33 of its 45 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

33 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Octo: An Open-Source Generalist Robot Policy20 May

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
20 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  7. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  8. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  11. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  12. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  13. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  14. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  15. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  16. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  17. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  18. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  19. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  20. 2023Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguardscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 52; 2026: 51.

106 papers.

Sort:

By area:

  1. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  4. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  5. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  6. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  7. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  8. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  9. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  10. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
  11. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  12. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
38 more
  1. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  2. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  3. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  4. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 47
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  7. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  9. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  10. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  11. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  12. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
  13. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 26
  14. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 25
  15. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  16. 2025ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesiscited by 22
  17. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  18. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correctioncited by 21
  19. 2026Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stackcited by 20
  20. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  21. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  22. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
  23. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16
  24. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  25. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  26. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 14
  27. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  28. 2025Human-assisted Robotic Policy Refinement via Action Preference Optimizationcited by 13
  29. 2025VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Searchcited by 12
  30. 2026VLANeXt: Recipes for Building Strong VLA Modelscited by 12
  31. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  32. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  33. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  34. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  35. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  36. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  37. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusioncited by 8
  38. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models
  2. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  3. 2026AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies
  4. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  5. 2026Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment
  6. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models
  7. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization
  8. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review