# Towards Scene Understanding with Detailed 3D Object Representations

arXiv: https://arxiv.org/abs/1411.5935 · HTML: https://mlsanity.com/paper/1411.5935/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2014.

## Built on by (most cited first, 9 of 9)

- [Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image](https://mlsanity.com/paper/2111.03098/) (2021)
- [HoW-3D: Holistic 3D Wireframe Perception from a Single Image](https://mlsanity.com/paper/2208.06999/) (2022)
- [Survey and Systematization of 3D Object Detection Models and Methods](https://mlsanity.com/paper/2201.09354/) (2022)
- [WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion](https://mlsanity.com/paper/2403.19022/) (2024)
- [Probabilistic Vehicle Reconstruction Using a Multi-Task CNN](https://mlsanity.com/paper/2102.10681/) (2021)
- [Provident Vehicle Detection at Night for Advanced Driver Assistance Systems](https://mlsanity.com/paper/2107.11302/) (2021)
- [GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion](https://mlsanity.com/paper/2605.12957/) (2026)
- [Pose Estimation and 3D Reconstruction of Vehicles from Stereo-Images Using a Subcategory-Aware Shape Prior](https://mlsanity.com/paper/2107.10898/) (2021)
- [SG-DOR: Learning Scene Graphs with Direction-Conditioned Occlusion Reasoning for Pepper Plants](https://mlsanity.com/paper/2603.06512/) (2026)

## Abstract (arXiv)

Current approaches to semantic image and scene understanding typically employ rather simple object representations such as 2D or 3D bounding boxes. While such coarse models are robust and allow for reliable object detection, they discard much of the information about objects' 3D shape and pose, and thus do not lend themselves well to higher-level reasoning. Here, we propose to base scene understanding on a high-resolution object representation. An object class - in our case cars - is modeled as a deformable 3D wireframe, which enables fine-grained modeling at the level of individual vertices and faces. We augment that model to explicitly include vertex-level occlusion, and embed all instances in a common coordinate frame, in order to infer and exploit object-object interactions. Specifically, from a single view we jointly estimate the shapes and poses of multiple objects in a common 3D frame. A ground plane in that frame is estimated by consensus among different objects, which significantly stabilizes monocular 3D pose estimation. The fine-grained model, in conjunction with the explicit 3D scene model, further allows one to infer part-level occlusions between the modeled objects, as well as occlusions by other, unmodeled scene elements. To demonstrate the benefits of such detailed object class models in the context of scene understanding we systematically evaluate our approach on the challenging KITTI street scene dataset. The experiments show that the model's ability to utilize image evidence at the level of individual parts improves monocular 3D pose estimation w.r.t. both location and (continuous) viewpoint.