# Virtual-to-Real: Learning to Control in Visual Semantic Segmentation

arXiv: https://arxiv.org/abs/1802.00285 · HTML: https://mlsanity.com/paper/1802.00285/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2018.

## Built on by (most cited first, 12 of 12)

- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Polybot: Training One Policy Across Robots While Embracing Variability](https://mlsanity.com/paper/2307.03719/) (2023)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Efficient Self-Ensemble for Semantic Segmentation](https://mlsanity.com/paper/2111.13280/) (2021)
- [RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking](https://mlsanity.com/paper/2304.03623/) (2023)
- [Coordinate-Aligned Multi-Camera Collaboration for Active Multi-Object Tracking](https://mlsanity.com/paper/2202.10881/) (2022)
- [How To Train Your HERON](https://mlsanity.com/paper/2102.10357/) (2021)
- [Investigation of Factorized Optical Flows as Mid-Level Representations](https://mlsanity.com/paper/2203.04927/) (2022)
- [Adapted Human Pose: Monocular 3D Human Pose Estimation with Zero Real 3D Pose Data](https://mlsanity.com/paper/2105.10837/) (2021)
- [Virtual Guidance as a Mid-level Representation for Navigation with Augmented Reality](https://mlsanity.com/paper/2303.02731/) (2023)
- [Visual Forecasting as a Mid-level Representation for Avoidance](https://mlsanity.com/paper/2310.07724/) (2023)

## Abstract (arXiv)

Collecting training data from the physical world is usually time-consuming and even dangerous for fragile robots, and thus, recent advances in robot learning advocate the use of simulators as the training platform. Unfortunately, the reality gap between synthetic and real visual data prohibits direct migration of the models trained in virtual worlds to the real world. This paper proposes a modular architecture for tackling the virtual-to-real problem. The proposed architecture separates the learning model into a perception module and a control policy module, and uses semantic image segmentation as the meta representation for relating these two modules. The perception module translates the perceived RGB image to semantic image segmentation. The control policy module is implemented as a deep reinforcement learning agent, which performs actions based on the translated image segmentation. Our architecture is evaluated in an obstacle avoidance task and a target following task. Experimental results show that our architecture significantly outperforms all of the baseline methods in both virtual and real environments, and demonstrates a faster learning curve than them. We also present a detailed analysis for a variety of variant configurations, and validate the transferability of our modular architecture.