# Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter

arXiv: https://arxiv.org/abs/2210.17325 · HTML: https://mlsanity.com/paper/2210.17325/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Learning Affordance Landscapes for Interaction Exploration in 3D Environments](https://mlsanity.com/paper/2008.09241/) (2020)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [ILabel: Interactive Neural Scene Labelling](https://mlsanity.com/paper/2111.14637/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Building an Affordances Map with Interactive Perception](https://mlsanity.com/paper/1903.04413/) (2019)
- [NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction](https://mlsanity.com/paper/2109.13859/) (2021)
- [Bootstrapping Robotic Ecological Perception from a Limited Set of Hypotheses Through Interactive Perception](https://mlsanity.com/paper/1901.10968/) (2019)

## Built on by (most cited first, 4 of 4)

- [VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations](https://mlsanity.com/paper/2305.12427/) (2023)
- [Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Review](https://mlsanity.com/paper/2402.11141/) (2024)
- [LiLMaps: Learnable Implicit Language Maps](https://mlsanity.com/paper/2501.03304/) (2025)
- [Diagnosing and Dynamically Filtering Occupancy World Models for Active Mapping](https://mlsanity.com/paper/2609.06820/) (2026)

## Abstract (arXiv)

Neural fields can be trained from scratch to represent the shape and appearance of 3D scenes efficiently. It has also been shown that they can densely map correlated properties such as semantics, via sparse interactions from a human labeller. In this work, we show that a robot can densely annotate a scene with arbitrary discrete or continuous physical properties via its own fully-autonomous experimental interactions, as it simultaneously scans and maps it with an RGB-D camera. A variety of scene interactions are possible, including poking with force sensing to determine rigidity, measuring local material type with single-pixel spectroscopy or predicting force distributions by pushing. Sparse experimental interactions are guided by entropy to enable high efficiency, with tabletop scene properties densely mapped from scratch in a few minutes from a few tens of interactions.