SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
Not on the map. Why?
Abstract (arXiv)
Robotic agents trained using reinforcement learning have the problem of taking unreliable actions in an out-of-distribution (OOD) state. Agents can easily become OOD in real-world environments because it is almost impossible for them to visit and learn the entire state space during training. Unfortunately, unreliable actions do not ensure that agents perform their original tasks successfully. Therefore, agents should be able to recognize whether they are in OOD states and learn how to return to the learned state distribution rather than continue to take unreliable actions. In this study, we propose a novel method for retraining agents to recover from OOD situations in a self-supervised manner when they fall into OOD states. Our in-depth experimental results demonstrate that our method substantially improves the agent's ability to recover from OOD situations in terms of sample efficiency and restoration of the performance for the original tasks. Moreover, we show that our method can retrain the agent to recover from OOD situations even when in-distribution states are difficult to visit through exploration.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 7 Nov 2023.
Builds on
25 of 34 references.
References · 25
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2016Concrete Problems in AI Safetycited by 1,642
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
- 2019Solving Rubik's Cube with a Robot Handcited by 734
- 2019Behavior Regularized Offline Reinforcement Learningcited by 662
- 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
13 more
- 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
- 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
- 2020Bayesian Neural Networks: An Introduction and Surveycited by 131
- 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
- 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 90
- 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 88
- 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
- 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
- 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 33
- 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
- 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
- 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 22
- 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 19
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.