# An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments

arXiv: https://arxiv.org/abs/2111.14666 · HTML: https://mlsanity.com/paper/2111.14666/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers](https://mlsanity.com/paper/2006.13916/) (2020)
- [MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation](https://mlsanity.com/paper/2012.03912/) (2020)
- [RobustNav: Towards Benchmarking Robustness in Embodied Navigation](https://mlsanity.com/paper/2106.04531/) (2021)
- [PyRobot: An Open-source Robotics Framework for Research and Benchmarking](https://mlsanity.com/paper/1906.08236/) (2019)
- [EgoMap: Projective mapping and structured egocentric memory for Deep RL](https://mlsanity.com/paper/2002.02286/) (2020)
- [DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning](https://mlsanity.com/paper/1909.02982/) (2019)
- [Adversarial Discriminative Sim-to-real Transfer of Visuo-motor Policies](https://mlsanity.com/paper/1709.05746/) (2017)
- [Deep Reinforcement Learning on a Budget: 3D Control and Reasoning Without a Supercomputer](https://mlsanity.com/paper/1904.01806/) (2019)

## Built on by (most cited first, 6 of 6)

- [Phone2Proc: Bringing Robust Robots Into Our Chaotic World](https://mlsanity.com/paper/2212.04819/) (2022)
- [Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction](https://mlsanity.com/paper/2306.03857/) (2023)
- [Learning to navigate efficiently and precisely in real environments](https://mlsanity.com/paper/2401.14349/) (2024)
- [Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach](https://mlsanity.com/paper/2503.08306/) (2025)
- [Multi-Object Navigation in real environments using hybrid policies](https://mlsanity.com/paper/2401.13800/) (2024)
- [Synthetic vs. Real Training Data for Visual Navigation](https://mlsanity.com/paper/2509.11791/) (2025)

## Abstract (arXiv)

Visual navigation by mobile robots is classically tackled through SLAM plus optimal planning, and more recently through end-to-end training of policies implemented as deep networks. While the former are often limited to waypoint planning, but have proven their efficiency even on real physical environments, the latter solutions are most frequently employed in simulation, but have been shown to be able learn more complex visual reasoning, involving complex semantical regularities. Navigation by real robots in physical environments is still an open problem. End-to-end training approaches have been thoroughly tested in simulation only, with experiments involving real robots being restricted to rare performance evaluations in simplified laboratory conditions. In this work we present an in-depth study of the performance and reasoning capacities of real physical agents, trained in simulation and deployed to two different physical environments. Beyond benchmarking, we provide insights into the generalization capabilities of different agents training in different conditions. We visualize sensor usage and the importance of the different types of signals. We show, that for the PointGoal task, an agent pre-trained on wide variety of tasks and fine-tuned on a simulated version of the target environment can reach competitive performance without modelling any sim2real transfer, i.e. by deploying the trained agent directly from simulation to a real physical robot.