# Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation

arXiv: https://arxiv.org/abs/2107.13545 · HTML: https://mlsanity.com/paper/2107.13545/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 28 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [BADGR: An Autonomous Self-Supervised Learning-Based Navigation System](https://mlsanity.com/paper/2002.05700/) (2020)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation](https://mlsanity.com/paper/1709.10489/) (2017)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias](https://mlsanity.com/paper/1807.07049/) (2018)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation](https://mlsanity.com/paper/2103.10534/) (2021)
- [PyRobot: An Open-source Robotics Framework for Research and Benchmarking](https://mlsanity.com/paper/1906.08236/) (2019)
- [Deep Predictive Policy Training using Reinforcement Learning](https://mlsanity.com/paper/1703.00727/) (2017)
- [Spatial Action Maps for Mobile Manipulation](https://mlsanity.com/paper/2004.09141/) (2020)
- [Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning](https://mlsanity.com/paper/1703.03078/) (2017)
- [Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learning](https://mlsanity.com/paper/2101.05325/) (2021)
- [Path Integral Guided Policy Search](https://mlsanity.com/paper/1610.00529/) (2016)
- [Ecological Reinforcement Learning](https://mlsanity.com/paper/2006.12478/) (2020)
- [Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversal](https://mlsanity.com/paper/1807.05211/) (2018)
- [Whole-Body Control of a Mobile Manipulator using End-to-End Reinforcement Learning](https://mlsanity.com/paper/2003.02637/) (2020)
- [AutOTranS: an Autonomous Open World Transportation System](https://mlsanity.com/paper/1810.03400/) (2018)

## Built on by (most cited first, 30 of 35)

- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning](https://mlsanity.com/paper/2310.15145/) (2023)
- [Multi-skill Mobile Manipulation for Object Rearrangement](https://mlsanity.com/paper/2209.02778/) (2022)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [Learning Generalizable Feature Fields for Mobile Manipulation](https://mlsanity.com/paper/2403.07563/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [Affordance-Guided Reinforcement Learning via Visual Prompting](https://mlsanity.com/paper/2407.10341/) (2024)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion](https://mlsanity.com/paper/2309.15459/) (2023)
- [Continuously Improving Mobile Manipulation with Autonomous Real-World RL](https://mlsanity.com/paper/2409.20568/) (2024)
- [RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World](https://mlsanity.com/paper/2412.00171/) (2024)
- [FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing](https://mlsanity.com/paper/2304.09831/) (2023)
- [Harmonic Mobile Manipulation](https://mlsanity.com/paper/2312.06639/) (2023)
- [SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training](https://mlsanity.com/paper/2506.04147/) (2025)
- [WildLMa: Long Horizon Loco-Manipulation in the Wild](https://mlsanity.com/paper/2411.15131/) (2024)
- [Versatile Loco-Manipulation through Flexible Interlimb Coordination](https://mlsanity.com/paper/2506.07876/) (2025)
- [Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation](https://mlsanity.com/paper/2504.09532/) (2025)
- [FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination](https://mlsanity.com/paper/2512.04381/) (2025)
- [Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower](https://mlsanity.com/paper/2204.02390/) (2022)
- [CARoL: Context-aware Adaptation for Robot Learning](https://mlsanity.com/paper/2506.07006/) (2025)
- [HiBerNAC: Hierarchical Brain-inspired Robotic Neural Agent Collective for Disentangling Complex Manipulation](https://mlsanity.com/paper/2506.08296/) (2025)
- [SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding](https://mlsanity.com/paper/2512.12842/) (2025)
- [Reset-Free Reinforcement Learning for Real-World Agile Driving: An Empirical Study](https://mlsanity.com/paper/2604.07672/) (2026)
- [Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models](https://mlsanity.com/paper/2609.32069/) (2026)
- [Adaptive Aggregation for Safety-Critical Control](https://mlsanity.com/paper/2302.03586/) (2023)
- [Non-Prehensile Aerial Manipulation using Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2407.00889/) (2024)
- [A Graph-Based Reinforcement Learning Approach with Frontier Potential Based Reward for Safe Cluttered Environment Exploration](https://mlsanity.com/paper/2504.11907/) (2025)

## Abstract (arXiv)

We study how robots can autonomously learn skills that require a combination of navigation and grasping. While reinforcement learning in principle provides for automated robotic skill learning, in practice reinforcement learning in the real world is challenging and often requires extensive instrumentation and supervision. Our aim is to devise a robotic reinforcement learning system for learning navigation and manipulation together, in an autonomous way without human intervention, enabling continual learning under realistic assumptions. Our proposed system, ReLMM, can learn continuously on a real-world platform without any environment instrumentation, without human intervention, and without access to privileged information, such as maps, objects positions, or a global view of the environment. Our method employs a modularized policy with components for manipulation and navigation, where manipulation policy uncertainty drives exploration for the navigation controller, and the manipulation module provides rewards for navigation. We evaluate our method on a room cleanup task, where the robot must navigate to and pick up items scattered on the floor. After a grasp curriculum training phase, ReLMM can learn navigation and grasping together fully automatically, in around 40 hours of autonomous real-world training.