# Invariance is Key to Generalization: Examining the Role of Representation in Sim-to-Real Transfer for Visual Navigation

arXiv: https://arxiv.org/abs/2310.15020 · HTML: https://mlsanity.com/paper/2310.15020/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [VR-Goggles for Robots: Real-to-sim Domain Adaptation for Visual Control](https://mlsanity.com/paper/1802.00265/) (2018)
- [Bi-directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents](https://mlsanity.com/paper/2011.12421/) (2020)
- [From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence](https://mlsanity.com/paper/2110.15245/) (2021)
- [Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation](https://mlsanity.com/paper/2011.06698/) (2020)
- [Learning to Navigate Using Mid-Level Visual Priors](https://mlsanity.com/paper/1912.11121/) (2019)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)

## Built on by (most cited first, 4 of 4)

- [Natural Language Can Help Bridge the Sim2Real Gap](https://mlsanity.com/paper/2405.10020/) (2024)
- [Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulation](https://mlsanity.com/paper/2503.11999/) (2025)
- [Towards Embodiment Scaling Laws in Robot Locomotion](https://mlsanity.com/paper/2505.05753/) (2025)
- [Understanding the Role of Invariance in Transfer Learning](https://mlsanity.com/paper/2407.04325/) (2024)

## Abstract (arXiv)

The data-driven approach to robot control has been gathering pace rapidly, yet generalization to unseen task domains remains a critical challenge. We argue that the key to generalization is representations that are (i) rich enough to capture all task-relevant information and (ii) invariant to superfluous variability between the training and the test domains. We experimentally study such a representation -- containing both depth and semantic information -- for visual navigation and show that it enables a control policy trained entirely in simulated indoor scenes to generalize to diverse real-world environments, both indoors and outdoors. Further, we show that our representation reduces the A-distance between the training and test domains, improving the generalization error bound as a result. Our proposed approach is scalable: the learned policy improves continuously, as the foundation models that it exploits absorb more diverse data during pre-training.