# Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D Cameras

arXiv: https://arxiv.org/abs/1703.08866 · HTML: https://mlsanity.com/paper/1703.08866/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2017.

## Built on by (most cited first, 28 of 28)

- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion](https://mlsanity.com/paper/2306.04633/) (2023)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision](https://mlsanity.com/paper/2303.03361/) (2023)
- [LaTeRF: Label and Text Driven Object Radiance Fields](https://mlsanity.com/paper/2207.01583/) (2022)
- [N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields](https://mlsanity.com/paper/2403.10997/) (2024)
- [DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation](https://mlsanity.com/paper/2411.04999/) (2024)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Semantic Visual Simultaneous Localization and Mapping: A Survey](https://mlsanity.com/paper/2209.06428/) (2022)
- [Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation](https://mlsanity.com/paper/2303.08401/) (2023)
- [Deep feature selection-and-fusion for RGB-D semantic segmentation](https://mlsanity.com/paper/2105.04102/) (2021)
- [Learn2Agree: Fitting with Multiple Annotators without Objective Ground Truth](https://mlsanity.com/paper/2109.03596/) (2021)
- [Topological Semantic Mapping by Consolidation of Deep Visual Features](https://mlsanity.com/paper/2106.12709/) (2021)
- [Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields](https://mlsanity.com/paper/2107.06028/) (2021)
- [SeMLaPS: Real-time Semantic Mapping with Latent Prior Networks and Quasi-Planar Segmentation](https://mlsanity.com/paper/2306.16585/) (2023)
- [MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion](https://mlsanity.com/paper/2405.19921/) (2024)
- [Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes](https://mlsanity.com/paper/2111.11103/) (2021)
- [MVMO: A Multi-Object Dataset for Wide Baseline Multi-View Semantic Segmentation](https://mlsanity.com/paper/2205.15452/) (2022)
- [Multi-View Video-Based Learning: Leveraging Weak Labels for Frame-Level Perception](https://mlsanity.com/paper/2403.11616/) (2024)
- [Enhancing Screen Time Identification in Children with a Multi-View Vision Language Model and Screen Time Tracker](https://mlsanity.com/paper/2410.01966/) (2024)
- [Multi-View Projection for Unsupervised Domain Adaptation in 3D Semantic Segmentation](https://mlsanity.com/paper/2505.15545/) (2025)
- [Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videos](https://mlsanity.com/paper/2506.05546/) (2025)
- [Enhancing Human-Robot Collaboration: A Sim2Real Domain Adaptation Algorithm for Point Cloud Segmentation in Industrial Environments](https://mlsanity.com/paper/2506.09552/) (2025)
- [PanOVOcc: Panoramic Embodied Open-Vocabulary Occupancy Mapping with Long-term Spatial Voxel Memory](https://mlsanity.com/paper/2609.31716/) (2026)

## Abstract (arXiv)

Visual scene understanding is an important capability that enables robots to purposefully act in their environment. In this paper, we propose a novel approach to object-class segmentation from multiple RGB-D views using deep learning. We train a deep neural network to predict object-class semantics that is consistent from several view points in a semi-supervised way. At test time, the semantics predictions of our network can be fused more consistently in semantic keyframe maps than predictions of a network trained on individual views. We base our network architecture on a recent single-view deep learning approach to RGB and depth fusion for semantic object-class segmentation and enhance it with multi-scale loss minimization. We obtain the camera trajectory using RGB-D SLAM and warp the predictions of RGB-D images into ground-truth annotated frames in order to enforce multi-view consistency during training. At test time, predictions from multiple views are fused into keyframes. We propose and analyze several methods for enforcing multi-view consistency during training and testing. We evaluate the benefit of multi-view consistency training and demonstrate that pooling of deep features and fusion over multiple views outperforms single-view baselines on the NYUDv2 benchmark for semantic segmentation. Our end-to-end trained network achieves state-of-the-art performance on the NYUDv2 dataset in single-view segmentation as well as multi-view semantic fusion.