# CVSformer: Cross-View Synthesis Transformer for Semantic Scene Completion

arXiv: https://arxiv.org/abs/2307.07938 · HTML: https://mlsanity.com/paper/2307.07938/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 30 of 30)

- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Anisotropic Convolutional Networks for 3D Semantic Scene Completion](https://mlsanity.com/paper/2004.02122/) (2020)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation](https://mlsanity.com/paper/1803.10409/) (2018)
- [S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds](https://mlsanity.com/paper/2012.09242/) (2020)
- [OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.13540/) (2023)
- [Efficient Semantic Scene Completion Network with Spatial Group Convolution](https://mlsanity.com/paper/1907.05091/) (2019)
- [RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion](https://mlsanity.com/paper/1903.00620/) (2019)
- [Semantic Scene Completion via Integrating Instances and Scene in-the-Loop](https://mlsanity.com/paper/2104.03640/) (2021)
- [Two Stream 3D Semantic Scene Completion](https://mlsanity.com/paper/1804.03550/) (2018)
- [VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention](https://mlsanity.com/paper/2203.09704/) (2022)
- [SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation](https://mlsanity.com/paper/2012.04512/) (2020)
- [Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion](https://mlsanity.com/paper/1908.00382/) (2019)
- [Depth Based Semantic Scene Completion with Position Importance Aware Loss](https://mlsanity.com/paper/2001.10709/) (2020)
- [ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image](https://mlsanity.com/paper/1909.01106/) (2019)
- [Deformable 3D Convolution for Video Super-Resolution](https://mlsanity.com/paper/2004.02803/) (2020)
- [Attention-based Multi-modal Fusion Network for Semantic Scene Completion](https://mlsanity.com/paper/2003.13910/) (2020)
- [View-volume Network for Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1806.05361/) (2018)
- [MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data](https://mlsanity.com/paper/2104.10772/) (2021)
- [3D Semantic Scene Completion from a Single Depth Image using Adversarial Training](https://mlsanity.com/paper/1905.06231/) (2019)
- [Adversarial Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1810.10901/) (2018)
- [IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement](https://mlsanity.com/paper/2106.15413/) (2021)
- [Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors](https://mlsanity.com/paper/2111.13309/) (2021)
- [3D Gated Recurrent Fusion for Semantic Scene Completion](https://mlsanity.com/paper/2002.07269/) (2020)

## Built on by (most cited first, 4 of 4)

- [Enhancing Object Coherence in Layout-to-Image Synthesis](https://mlsanity.com/paper/2311.10522/) (2023)
- [Unleashing Network Potentials for Semantic Scene Completion](https://mlsanity.com/paper/2403.07560/) (2024)
- [Deep Learning for 3D Point Cloud Enhancement: A Survey](https://mlsanity.com/paper/2411.00857/) (2024)
- [Point-based Instance Completion with Scene Constraints](https://mlsanity.com/paper/2504.05698/) (2025)

## Abstract (arXiv)

Semantic scene completion (SSC) requires an accurate understanding of the geometric and semantic relationships between the objects in the 3D scene for reasoning the occluded objects. The popular SSC methods voxelize the 3D objects, allowing the deep 3D convolutional network (3D CNN) to learn the object relationships from the complex scenes. However, the current networks lack the controllable kernels to model the object relationship across multiple views, where appropriate views provide the relevant information for suggesting the existence of the occluded objects. In this paper, we propose Cross-View Synthesis Transformer (CVSformer), which consists of Multi-View Feature Synthesis and Cross-View Transformer for learning cross-view object relationships. In the multi-view feature synthesis, we use a set of 3D convolutional kernels rotated differently to compute the multi-view features for each voxel. In the cross-view transformer, we employ the cross-view fusion to comprehensively learn the cross-view relationships, which form useful information for enhancing the features of individual views. We use the enhanced features to predict the geometric occupancies and semantic labels of all voxels. We evaluate CVSformer on public datasets, where CVSformer yields state-of-the-art results.