SIM2REALVIZ: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation
Not on the map. Why?
Abstract (arXiv)
The Robotics community has started to heavily rely on increasingly realistic 3D simulators for large-scale training of robots on massive amounts of data. But once robots are deployed in the real world, the simulation gap, as well as changes in the real world (e.g. lights, objects displacements) lead to errors. In this paper, we introduce Sim2RealViz, a visual analytics tool to assist experts in understanding and reducing this gap for robot ego-pose estimation tasks, i.e. the estimation of a robot's position using trained models. Sim2RealViz displays details of a given model and the performance of its instances in both simulation and real-world. Experts can identify environment differences that impact model predictions at a given location and explore through direct interactions with the model hypothesis to fix it. We detail the design of the tool, and case studies related to the exploit of the regression to the mean bias and how it can be addressed, and how models are perturbed by the vanish of landmarks such as bikes.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 24 Sep 2021.
Builds on
31 of 46 references.
References · 31
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
- 2013Visualizing and Understanding Convolutional Networkscited by 2,278
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
- 2017Adversarial Discriminative Domain Adaptationcited by 1,309
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2014Striving for Simplicity: The All Convolutional Netcited by 983
- 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 307
19 more
- 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292
- 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
- 2019Active Domain Randomizationcited by 161
- 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
- 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
- 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
- 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
- 2019Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizationscited by 72
- 2021On the Limits of Pseudo Ground Truth in Visual Camera Re-localisationcited by 63
- 2016Towards Better Analysis of Deep Convolutional Neural Networkscited by 51
- 2017Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulationcited by 35
- 2020Analyzing the Noise Robustness of Deep Neural Networkscited by 31
- 2015Adapting Deep Visuomotor Representations with Weak Pairwise Constraintscited by 25
- 2019DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learningcited by 12
- 2017Diabetic Retinopathy Detection via Deep Convolutional Networks for Discriminative Localization and Visual Explanationcited by 11
- 2018Improved Techniques for Adversarial Discriminative Domain Adaptationcited by 10
- 2020A Visual Analytics Framework for Explaining and Diagnosing Transfer Learning Processescited by 9
- 2019Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigationcited by 6
- 2020Visualizing Transfer Learningcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.