# The Robotic Vision Scene Understanding Challenge

arXiv: https://arxiv.org/abs/2009.05246 · HTML: https://mlsanity.com/paper/2009.05246/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2020.

## Built on by (most cited first, 9 of 9)

- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenes](https://mlsanity.com/paper/2108.05246/) (2021)
- [Semantic Map-based Generation of Navigation Instructions](https://mlsanity.com/paper/2403.19603/) (2024)
- [Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics](https://mlsanity.com/paper/2304.12289/) (2023)
- [Tactile MNIST: Benchmarking Active Tactile Perception](https://mlsanity.com/paper/2506.06361/) (2025)
- [Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments](https://mlsanity.com/paper/2301.09854/) (2023)
- [3D Semantic MapNet: Building Maps for Multi-Object Re-Identification in 3D](https://mlsanity.com/paper/2403.13190/) (2024)
- [A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods](https://mlsanity.com/paper/2507.01143/) (2025)
- [Indoor Semantic Scene Understanding using Multi-modality Fusion](https://mlsanity.com/paper/2108.07616/) (2021)

## Abstract (arXiv)

Being able to explore an environment and understand the location and type of all objects therein is important for indoor robotic platforms that must interact closely with humans. However, it is difficult to evaluate progress in this area due to a lack of standardized testing which is limited due to the need for active robot agency and perfect object ground-truth. To help provide a standard for testing scene understanding systems, we present a new robot vision scene understanding challenge using simulation to enable repeatable experiments with active robot agency. We provide two challenging task types, three difficulty levels, five simulated environments and a new evaluation measure for evaluating 3D cuboid object maps. Our aim is to drive state-of-the-art research in scene understanding through enabling evaluation and comparison of active robotic vision systems.