Validate on Sim, Detect on Real -- Model Selection for Domain Randomization
Not on the map. Why?
Abstract (arXiv)
A practical approach to learning robot skills, often termed sim2real, is to train control policies in simulation and then deploy them on a real robot. Popular techniques to improve the sim2real transfer build on domain randomization (DR) -- training the policy on a diverse set of randomly generated domains with the hope of better generalization to the real world. Due to the large number of hyper-parameters in both the policy learning and DR algorithms, one often ends up with a large number of trained policies, where choosing the best policy among them demands costly evaluation on the real robot. In this work we ask - can we rank the policies without running them in the real world? Our main idea is that a predefined set of real world data can be used to evaluate all policies, using out-of-distribution detection (OOD) techniques. In a sense, this approach can be seen as a `unit test' to evaluate policies before any real world execution. However, we find that by itself, the OOD score can be inaccurate and very sensitive to the particular OOD method. Our main contribution is a simple-yet-effective policy score that combines OOD with an evaluation in simulation. We show that our score - VSDR - can significantly improve the accuracy of policy ranking without requiring additional real world data. We evaluate the effectiveness of VSDR on sim2real transfer in a robotic grasping task with image inputs. We extensively evaluate different DR parameters and OOD methods, and show that VSDR improves policy selection across the board. More importantly, our method achieves significantly better ranking, and uses significantly less data compared to baselines. Project website is available at https://sites.google.com/view/vsdr/home.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 1 Nov 2021.
Builds on
26 of 38 references.
References · 26
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2013Describing Textures in the Wildcited by 2,110
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2017Deep Reinforcement Learning that Matterscited by 727
- 2019Solving Rubik's Cube with a Robot Handcited by 699
- 2018Learning Dexterous In-Hand Manipulationcited by 643
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
- 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
- 2020Reinforcement Learning with Augmented Datacited by 452
14 more
- 2019Learning agile and dynamic motor skills for legged robotscited by 402
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
- 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
- 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
- 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
- 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 113
- 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
- 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
- 20163D Simulation for Robot Arm Control with Deep Q-Learningcited by 19
- 2017Sim2Real View Invariant Visual Servoing by Recurrent Controlcited by 19
- 2019Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisationcited by 9
- 2020A Self-Tuning Actor-Critic Algorithmcited by 6
Built on by
Citing papers per year: 2023: 3.
3 papers.