# Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.12974 · HTML: https://mlsanity.com/paper/2004.12974/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 30 of 31)

- [Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.01488/) (2023)
- [Autonomous Reinforcement Learning via Subgoal Curricula](https://mlsanity.com/paper/2107.12931/) (2021)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning](https://mlsanity.com/paper/2103.12726/) (2021)
- [Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2106.01404/) (2021)
- [Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration](https://mlsanity.com/paper/2109.08603/) (2021)
- [Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation](https://mlsanity.com/paper/2112.00597/) (2021)
- [Automating Reinforcement Learning with Example-based Resets](https://mlsanity.com/paper/2204.02041/) (2022)
- [Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization](https://mlsanity.com/paper/2110.04686/) (2021)
- [One After Another: Learning Incremental Skills for a Changing World](https://mlsanity.com/paper/2203.11176/) (2022)
- [Towards a Grounded Theory of Causation for Embodied AI](https://mlsanity.com/paper/2206.13973/) (2022)
- [A Mixture of Surprises for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2210.06702/) (2022)
- [Intrinsic Motivation in Dynamical Control Systems](https://mlsanity.com/paper/2301.00005/) (2022)
- [Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild](https://mlsanity.com/paper/2304.10888/) (2023)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2410.07877/) (2024)
- [Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery](https://mlsanity.com/paper/2204.13906/) (2022)
- [Continual Robot Learning using Self-Supervised Task Inference](https://mlsanity.com/paper/2309.04974/) (2023)
- [Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum](https://mlsanity.com/paper/2305.09943/) (2023)
- [Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment](https://mlsanity.com/paper/2504.14805/) (2025)
- [IPR-1: Interactive Physical Reasoner](https://mlsanity.com/paper/2511.15407/) (2025)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Self-Supervised Curriculum Generation for Autonomous Reinforcement Learning without Task-Specific Knowledge](https://mlsanity.com/paper/2311.09195/) (2023)
- [Situationally-Aware Dynamics Learning](https://mlsanity.com/paper/2505.19574/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2603.25464/) (2026)
- [Analyzing the Hidden Activations of Deep Policy Networks: Why Representation Matters](https://mlsanity.com/paper/2103.06398/) (2021)
- [Adaptation of Quadruped Robot Locomotion with Meta-Learning](https://mlsanity.com/paper/2107.03741/) (2021)
- [Efficient Exploration using Model-Based Quality-Diversity with Gradients](https://mlsanity.com/paper/2211.12610/) (2022)

## Abstract (arXiv)

Reinforcement learning provides a general framework for learning robotic skills while minimizing engineering effort. However, most reinforcement learning algorithms assume that a well-designed reward function is provided, and learn a single behavior for that single reward function. Such reward functions can be difficult to design in practice. Can we instead develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks? In this paper, we demonstrate that a recently proposed unsupervised skill discovery algorithm can be extended into an efficient off-policy method, making it suitable for performing unsupervised reinforcement learning in the real world. Firstly, we show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible. Secondly, we move beyond the simulation environments and evaluate the algorithm on real physical hardware. On quadrupeds, we observe that locomotion skills with diverse gaits and different orientations emerge without any rewards or demonstrations. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.