# Attention-based Multi-modal Fusion Network for Semantic Scene Completion

arXiv: https://arxiv.org/abs/2003.13910 · HTML: https://mlsanity.com/paper/2003.13910/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 28 of 28)

- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.13540/) (2023)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation](https://mlsanity.com/paper/2404.11958/) (2024)
- [SSC-RS: Elevate LiDAR Semantic Scene Completion with Representation Separation and BEV Fusion](https://mlsanity.com/paper/2306.15349/) (2023)
- [OccMamba: Semantic Occupancy Prediction with State Space Models](https://mlsanity.com/paper/2408.09859/) (2024)
- [Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspective](https://mlsanity.com/paper/2112.12925/) (2021)
- [Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2408.01366/) (2024)
- [2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision](https://mlsanity.com/paper/2310.12817/) (2023)
- [T2TD: Text-3D Generation Model based on Prior Knowledge Guidance](https://mlsanity.com/paper/2305.15753/) (2023)
- [Semantic Scene Completion with Cleaner Self](https://mlsanity.com/paper/2303.09977/) (2023)
- [CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion](https://mlsanity.com/paper/2211.13702/) (2022)
- [IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement](https://mlsanity.com/paper/2106.15413/) (2021)
- [Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors](https://mlsanity.com/paper/2111.13309/) (2021)
- [Label-efficient Semantic Scene Completion with Scribble Annotations](https://mlsanity.com/paper/2405.15170/) (2024)
- [Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2501.07260/) (2025)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [CVSformer: Cross-View Synthesis Transformer for Semantic Scene Completion](https://mlsanity.com/paper/2307.07938/) (2023)
- [V2VSSC: A 3D Semantic Scene Completion Benchmark for Perception with Vehicle to Vehicle Communication](https://mlsanity.com/paper/2402.04671/) (2024)
- [Deep Learning for 3D Point Cloud Enhancement: A Survey](https://mlsanity.com/paper/2411.00857/) (2024)
- [Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection](https://mlsanity.com/paper/2302.08670/) (2023)
- [Audio-video fusion strategies for active speaker detection in meetings](https://mlsanity.com/paper/2206.10411/) (2022)
- [NUAA-QMUL-AIIT at Memotion 3: Multi-modal Fusion with Squeeze-and-Excitation for Internet Meme Emotion Analysis](https://mlsanity.com/paper/2302.08326/) (2023)
- [Deflickering Vision-Based Occupancy Networks through Lightweight Spatio-Temporal Correlation](https://mlsanity.com/paper/2502.15438/) (2025)

## Abstract (arXiv)

This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task of inferring the occupancy and semantic labels of a volumetric 3D scene from single-view RGB-D images. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously via leveraging the experience of inferring 2D semantic segmentation from RGB-D images as well as the reliable depth cues in spatial dimension. It is achieved by employing a multi-modal fusion architecture boosted from 2D semantic segmentation and a 3D semantic completion network empowered by residual attention blocks. We validate our method on both the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results show that our method respectively achieves the gains of 2.5% and 2.6% on the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the state-of-the-art method.