# Object-Oriented Material Classification and 3D Clustering for Improved Semantic Perception and Mapping in Mobile Robots

arXiv: https://arxiv.org/abs/2407.06077 · HTML: https://mlsanity.com/paper/2407.06077/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Material Recognition in the Wild with the Materials in Context Database](https://mlsanity.com/paper/1412.0623/) (2014)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter](https://mlsanity.com/paper/1810.00818/) (2018)
- [How Can Robots Trust Each Other For Better Cooperation? A Relative Needs Entropy Based Robot-Robot Trust Assessment Model](https://mlsanity.com/paper/2105.07443/) (2021)
- [Online Object-Oriented Semantic Mapping and Map Updating](https://mlsanity.com/paper/2011.06895/) (2020)
- [Geometry-Informed Material Recognition](https://mlsanity.com/paper/1607.05338/) (2016)
- [Material Recognition from Local Appearance in Global Context](https://mlsanity.com/paper/1611.09394/) (2016)
- [A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition](https://mlsanity.com/paper/1703.04699/) (2017)
- [DGORL: Distributed Graph Optimization based Relative Localization of Multi-Robot Systems](https://mlsanity.com/paper/2210.01662/) (2022)
- [Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework using Visual and Depth Cues](https://mlsanity.com/paper/2003.06336/) (2020)
- [An Online Semantic Mapping System for Extending and Enhancing Visual SLAM](https://mlsanity.com/paper/2203.03944/) (2022)
- [SEAL: Simultaneous Exploration and Localization in Multi-Robot Systems](https://mlsanity.com/paper/2306.12623/) (2023)
- [Evaluating Deep Convolutional Neural Networks for Material Classification](https://mlsanity.com/paper/1703.04101/) (2017)
- [Multi-Robot Synergistic Localization in Dynamic Environments](https://mlsanity.com/paper/2206.03573/) (2022)

## Built on by (most cited first, 2 of 2)

- [SPACE: 3D Spatial Co-operation and Exploration Framework for Robust Mapping and Coverage with Multi-Robot Systems](https://mlsanity.com/paper/2411.02524/) (2024)
- [Sonify Anything: Towards Context-Aware Sonic Interactions in AR](https://mlsanity.com/paper/2508.01789/) (2025)

## Abstract (arXiv)

Classification of different object surface material types can play a significant role in the decision-making algorithms for mobile robots and autonomous vehicles. RGB-based scene-level semantic segmentation has been well-addressed in the literature. However, improving material recognition using the depth modality and its integration with SLAM algorithms for 3D semantic mapping could unlock new potential benefits in the robotics perception pipeline. To this end, we propose a complementarity-aware deep learning approach for RGB-D-based material classification built on top of an object-oriented pipeline. The approach further integrates the ORB-SLAM2 method for 3D scene mapping with multiscale clustering of the detected material semantics in the point cloud map generated by the visual SLAM algorithm. Extensive experimental results with existing public datasets and newly contributed real-world robot datasets demonstrate a significant improvement in material classification and 3D clustering accuracy compared to state-of-the-art approaches for 3D semantic scene mapping.