# Validate on Sim, Detect on Real -- Model Selection for Domain Randomization

arXiv: https://arxiv.org/abs/2111.00765 · HTML: https://mlsanity.com/paper/2111.00765/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Hyperparameter Selection for Offline Reinforcement Learning](https://mlsanity.com/paper/2007.09055/) (2020)
- [Off-Policy Evaluation via Off-Policy Classification](https://mlsanity.com/paper/1906.01624/) (2019)
- [3D Simulation for Robot Arm Control with Deep Q-Learning](https://mlsanity.com/paper/1609.03759/) (2016)
- [Sim2Real View Invariant Visual Servoing by Recurrent Control](https://mlsanity.com/paper/1712.07642/) (2017)
- [Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation](https://mlsanity.com/paper/1912.08324/) (2019)
- [A Self-Tuning Actor-Critic Algorithm](https://mlsanity.com/paper/2002.12928/) (2020)

## Built on by (most cited first, 3 of 3)

- [Marginalized Importance Sampling for Off-Environment Policy Evaluation](https://mlsanity.com/paper/2309.01807/) (2023)
- [GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2309.09051/) (2023)
- [GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2306.09872/) (2023)

## Abstract (arXiv)

A practical approach to learning robot skills, often termed sim2real, is to train control policies in simulation and then deploy them on a real robot. Popular techniques to improve the sim2real transfer build on domain randomization (DR) -- training the policy on a diverse set of randomly generated domains with the hope of better generalization to the real world. Due to the large number of hyper-parameters in both the policy learning and DR algorithms, one often ends up with a large number of trained policies, where choosing the best policy among them demands costly evaluation on the real robot. In this work we ask - can we rank the policies without running them in the real world? Our main idea is that a predefined set of real world data can be used to evaluate all policies, using out-of-distribution detection (OOD) techniques. In a sense, this approach can be seen as a `unit test' to evaluate policies before any real world execution. However, we find that by itself, the OOD score can be inaccurate and very sensitive to the particular OOD method. Our main contribution is a simple-yet-effective policy score that combines OOD with an evaluation in simulation. We show that our score - VSDR - can significantly improve the accuracy of policy ranking without requiring additional real world data. We evaluate the effectiveness of VSDR on sim2real transfer in a robotic grasping task with image inputs. We extensively evaluate different DR parameters and OOD methods, and show that VSDR improves policy selection across the board. More importantly, our method achieves significantly better ranking, and uses significantly less data compared to baselines. Project website is available at https://sites.google.com/view/vsdr/home.