# SIM2REALVIZ: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation

arXiv: https://arxiv.org/abs/2109.11801 · HTML: https://mlsanity.com/paper/2109.11801/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations](https://mlsanity.com/paper/1904.02323/) (2019)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation](https://mlsanity.com/paper/1710.06422/) (2017)
- [Analyzing the Noise Robustness of Deep Neural Networks](https://mlsanity.com/paper/2001.09395/) (2020)
- [Adapting Deep Visuomotor Representations with Weak Pairwise Constraints](https://mlsanity.com/paper/1511.07111/) (2015)
- [DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning](https://mlsanity.com/paper/1909.02982/) (2019)
- [Diabetic Retinopathy Detection via Deep Convolutional Networks for Discriminative Localization and Visual Explanation](https://mlsanity.com/paper/1703.10757/) (2017)
- [Improved Techniques for Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1809.03625/) (2018)
- [A Visual Analytics Framework for Explaining and Diagnosing Transfer Learning Processes](https://mlsanity.com/paper/2009.06876/) (2020)
- [Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation](https://mlsanity.com/paper/1904.03895/) (2019)

## Built on by (most cited first, 3 of 3)

- [CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulator](https://mlsanity.com/paper/2410.18238/) (2024)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [NeuroPAL: Punctuated Anytime Learning with Neuroevolution for Macromanagement in Starcraft: Brood War](https://mlsanity.com/paper/2506.10384/) (2025)

## Abstract (arXiv)

The Robotics community has started to heavily rely on increasingly realistic 3D simulators for large-scale training of robots on massive amounts of data. But once robots are deployed in the real world, the simulation gap, as well as changes in the real world (e.g. lights, objects displacements) lead to errors. In this paper, we introduce Sim2RealViz, a visual analytics tool to assist experts in understanding and reducing this gap for robot ego-pose estimation tasks, i.e. the estimation of a robot's position using trained models. Sim2RealViz displays details of a given model and the performance of its instances in both simulation and real-world. Experts can identify environment differences that impact model predictions at a given location and explore through direct interactions with the model hypothesis to fix it. We detail the design of the tool, and case studies related to the exploit of the regression to the mean bias and how it can be addressed, and how models are perturbed by the vanish of landmarks such as bikes.