# RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation

arXiv: https://arxiv.org/abs/2602.01789 · HTML: https://mlsanity.com/paper/2602.01789/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Builds on (most cited first, 30 of 42)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Rectified Flow: A Marginal Preserving Approach to Optimal Transport](https://mlsanity.com/paper/2209.14577/) (2022)
- [Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning](https://mlsanity.com/paper/2303.05479/) (2023)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [ALOHA Unleashed: A Simple Recipe for Robot Dexterity](https://mlsanity.com/paper/2410.13126/) (2024)
- [A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation](https://mlsanity.com/paper/2507.05331/) (2025)
- [Flow Matching Policy Gradients](https://mlsanity.com/paper/2507.21053/) (2025)
- [ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning](https://mlsanity.com/paper/2505.22094/) (2025)
- [From Imitation to Refinement -- Residual RL for Precise Assembly](https://mlsanity.com/paper/2407.16677/) (2024)

## Built on by (most cited first, 10 of 10)

- [You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector](https://mlsanity.com/paper/2603.15757/) (2026)
- [TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning](https://mlsanity.com/paper/2605.12236/) (2026)
- [REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning](https://mlsanity.com/paper/2603.13707/) (2026)
- [Reinforcement Learning for Flow-Matching Policies with Density Transport](https://mlsanity.com/paper/2606.08602/) (2026)
- [BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models](https://mlsanity.com/paper/2605.30226/) (2026)
- [Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience](https://mlsanity.com/paper/2606.27475/) (2026)
- [Source-Lifted Flow Matching for Intervenable Multimodal Imitation](https://mlsanity.com/paper/2607.10206/) (2026)
- [HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery](https://mlsanity.com/paper/2609.25558/) (2026)
- [The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models](https://mlsanity.com/paper/2609.27070/) (2026)
- [Data-Efficient Adaptation of a Driving VLA to Class 8 Trucks](https://mlsanity.com/paper/2609.38570/) (2026)

## Abstract (arXiv)

Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering(RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings when adapting pretrained base policies. Project website:https://weirdlabuw.github.io/rfs.