# DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding

arXiv: https://arxiv.org/abs/1603.04922 · HTML: https://mlsanity.com/paper/1603.04922/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2016.

## Built on by (most cited first, 11 of 11)

- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization](https://mlsanity.com/paper/2108.10743/) (2021)
- [Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images](https://mlsanity.com/paper/2105.02047/) (2021)
- [ARM3D: Attention-based relation module for indoor 3D object detection](https://mlsanity.com/paper/2202.09715/) (2022)
- [Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects](https://mlsanity.com/paper/2205.02714/) (2022)
- [ROOT: VLM based System for Indoor Scene Understanding and Beyond](https://mlsanity.com/paper/2411.15714/) (2024)
- [DisARM: Displacement Aware Relation Module for 3D Detection](https://mlsanity.com/paper/2203.01152/) (2022)
- [Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding](https://mlsanity.com/paper/2311.06694/) (2023)
- [Robust Shape Fitting for 3D Scene Abstraction](https://mlsanity.com/paper/2403.10452/) (2024)
- [3DRM:Pair-wise relation module for 3D object detection](https://mlsanity.com/paper/2202.09721/) (2022)
- [CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection](https://mlsanity.com/paper/2209.06641/) (2022)

## Abstract (arXiv)

While deep neural networks have led to human-level performance on computer vision tasks, they have yet to demonstrate similar gains for holistic scene understanding. In particular, 3D context has been shown to be an extremely important cue for scene understanding - yet very little research has been done on integrating context information with deep models. This paper presents an approach to embed 3D context into the topology of a neural network trained to perform holistic scene understanding. Given a depth image depicting a 3D scene, our network aligns the observed scene with a predefined 3D scene template, and then reasons about the existence and location of each object within the scene template. In doing so, our model recognizes multiple objects in a single forward pass of a 3D convolutional neural network, capturing both global scene and local object information simultaneously. To create training data for this 3D network, we generate partly hallucinated depth images which are rendered by replacing real objects with a repository of CAD models of the same object category. Extensive experiments demonstrate the effectiveness of our algorithm compared to the state-of-the-arts. Source code and data are available at http://deepcontext.cs.princeton.edu.