# Uncertainty Aware System Identification with Universal Policies

arXiv: https://arxiv.org/abs/2202.05844 · HTML: https://mlsanity.com/paper/2202.05844/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators](https://mlsanity.com/paper/1906.01728/) (2019)
- [Sim-to-Real Transfer for Biped Locomotion](https://mlsanity.com/paper/1903.01390/) (2019)
- [Auto-Tuned Sim-to-Real Transfer](https://mlsanity.com/paper/2104.07662/) (2021)
- [Data-efficient Domain Randomization with Bayesian Optimization](https://mlsanity.com/paper/2003.02471/) (2020)
- [Policy Transfer with Strategy Optimization](https://mlsanity.com/paper/1810.05751/) (2018)
- [Emulation of physical processes with Emukit](https://mlsanity.com/paper/2110.13293/) (2021)
- [TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer](https://mlsanity.com/paper/1907.11200/) (2019)
- [Unscented Bayesian Optimization for Safe Robot Grasping](https://mlsanity.com/paper/1603.02038/) (2016)
- [Bayesian Policy Optimization for Model Uncertainty](https://mlsanity.com/paper/1810.01014/) (2018)

## Built on by (most cited first, 2 of 2)

- [LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World](https://mlsanity.com/paper/2409.19617/) (2024)
- [Domains as Objectives: Domain-Uncertainty-Aware Policy Optimization through Explicit Multi-Domain Convex Coverage Set Learning](https://mlsanity.com/paper/2410.04719/) (2024)

## Abstract (arXiv)

Sim2real transfer is primarily concerned with transferring policies trained in simulation to potentially noisy real world environments. A common problem associated with sim2real transfer is estimating the real-world environmental parameters to ground the simulated environment to. Although existing methods such as Domain Randomisation (DR) can produce robust policies by sampling from a distribution of parameters during training, there is no established method for identifying the parameters of the corresponding distribution for a given real-world setting. In this work, we propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies across the full range of environmental parameters and then subsequently employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion. Such policy-driven grounding is expected to be more efficient as it estimates only task-relevant sets of parameters. Further, we also account for the estimation uncertainties in the search process to produce policies that are robust against both aleatoric and epistemic uncertainties. We empirically evaluate our approach in a range of noisy, continuous control environments, and show its improved performance compared to competing baselines.