mlsanityData to 7 Oct 2026

Steering Your Diffusion Policy with Latent Space Reinforcement Learning

arXiv 2506.15799 ↗First posted 18 Jun 2025cs.RO cs.LGcited by 149

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Robotic control policies learned from human demonstrations have achieved impressive results in many real-world applications. However, in scenarios where initial performance is not satisfactory, as is often the case in novel open-world settings, such behavioral cloning (BC)-learned policies typically require collecting additional human demonstrations to further improve their behavior -- an expensive and time-consuming process. In contrast, reinforcement learning (RL) holds the promise of enabling autonomous online policy improvement, but often falls short of achieving this due to the large number of samples it typically requires. In this work we take steps towards enabling fast autonomous adaptation of BC-trained policies via efficient real-world RL. Focusing in particular on diffusion policies -- a state-of-the-art BC methodology -- we propose diffusion steering via reinforcement learning (DSRL): adapting the BC policy by running RL over its latent-noise space. We show that DSRL is highly sample efficient, requires only black-box access to the BC policy, and enables effective real-world autonomous policy improvement. Furthermore, DSRL avoids many of the challenges associated with finetuning diffusion policies, obviating the need to modify the weights of the base policy at all. We demonstrate DSRL on simulated benchmarks, real-world robotic tasks, and for adapting pretrained generalist policies, illustrating its sample efficiency and effective performance at real-world policy improvement.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 88 of its 94 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

88 of 94 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  2. 2025Flow Q-Learning4 Feb
  3. 2024The Ingredients for Robotic Diffusion Transformers14 Oct
  4. 2024Diffusion Policy Policy Optimization1 Sep
  5. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  6. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  7. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  8. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  9. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  10. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  11. 2024Simple Hierarchical Planning with Diffusion5 Jan
  12. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
15 more
  1. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  2. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration11 Oct
  3. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  4. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  5. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  6. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  7. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  9. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  13. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  14. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  15. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 61

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  11. 2015Continuous control with deep reinforcement learningcited by 3,279
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
38 more
  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
  13. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  14. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  15. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  16. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  17. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  18. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  19. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  20. 2024Flow Matching Guide and Codecited by 291
  21. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  22. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  23. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  24. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  25. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  26. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  27. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  28. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  29. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  30. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 128
  31. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  32. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  33. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  34. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  35. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  36. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  37. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  38. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 125.

149 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
  4. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  5. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  6. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  7. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  8. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  9. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  10. 2026Q-learning with Adjoint Matchingcited by 33
  11. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  12. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 26
38 more
  1. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  2. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  3. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  4. 2026Action-to-Action Flow Matchingcited by 15
  5. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  6. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 14
  7. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  8. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  9. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  10. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  11. 2025Multi-agent Coordination via Flow Matchingcited by 9
  12. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 8
  13. 2025Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusioncited by 8
  14. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  15. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  16. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  17. 2026RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Trainingcited by 7
  18. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  19. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  20. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  21. 2026Green-VLA: Staged Vision-Language-Action Model for Generalist Robotscited by 6
  22. 2026OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policiescited by 6
  23. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  24. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  25. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 5
  26. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  27. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptationcited by 5
  28. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  29. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 5
  30. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  31. 2026Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Modelscited by 4
  32. 2026FASTER: Value-Guided Sampling for Fast RLcited by 4
  33. 2026WarmPrior: Straightening Flow-Matching Policies with Temporal Priorscited by 4
  34. 2026TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulationcited by 4
  35. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  36. 2026WCM: A World Critic Model for Vision-Language-Action Reinforcement Learningcited by 4
  37. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  38. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control
  2. 2026Performance-Preserving Online Adaptation in Social Navigation via Diffusion Steering ↗
  3. 2026A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer
  4. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
  5. 2026Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies
  6. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Models
  7. 2025Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies
  8. 2025DiWA: Diffusion Policy Adaptation with World Models