# DirectShape: Direct Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation

arXiv: https://arxiv.org/abs/1904.10097 · HTML: https://mlsanity.com/paper/1904.10097/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2019.

## Built on by (most cited first, 11 of 11)

- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)
- [Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild](https://mlsanity.com/paper/2101.06860/) (2021)
- [3D Object Detection with a Self-supervised Lidar Scene Flow Backbone](https://mlsanity.com/paper/2205.00705/) (2022)
- [DirectTracker: 3D Multi-Object Tracking Using Direct Image Alignment and Photometric Bundle Adjustment](https://mlsanity.com/paper/2209.14965/) (2022)
- [Plausibility Verification For 3D Object Detectors Using Energy-Based Optimization](https://mlsanity.com/paper/2211.05233/) (2022)
- [Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos](https://mlsanity.com/paper/2309.15703/) (2023)
- [Category-Level Pose Retrieval with Contrastive Features Learnt with Occlusion Augmentation](https://mlsanity.com/paper/2208.06195/) (2022)
- [ShapeICP: Iterative Category-level Object Pose and Shape Estimation from Depth](https://mlsanity.com/paper/2408.13147/) (2024)
- [View-to-Label: Multi-View Consistency for Self-Supervised 3D Object Detection](https://mlsanity.com/paper/2305.17972/) (2023)
- [Globally Optimal Pose from Orthographic Silhouettes](https://mlsanity.com/paper/2604.09199/) (2026)

## Abstract (arXiv)

Scene understanding from images is a challenging problem encountered in autonomous driving. On the object level, while 2D methods have gradually evolved from computing simple bounding boxes to delivering finer grained results like instance segmentations, the 3D family is still dominated by estimating 3D bounding boxes. In this paper, we propose a novel approach to jointly infer the 3D rigid-body poses and shapes of vehicles from a stereo image pair using shape priors. Unlike previous works that geometrically align shapes to point clouds from dense stereo reconstruction, our approach works directly on images by combining a photometric and a silhouette alignment term in the energy function. An adaptive sparse point selection scheme is proposed to efficiently measure the consistency with both terms. In experiments, we show superior performance of our method on 3D pose and shape estimation over the previous geometric approach and demonstrate that our method can also be applied as a refinement step and significantly boost the performances of several state-of-the-art deep learning based 3D object detectors. All related materials and demonstration videos are available at the project page https://vision.in.tum.de/research/vslam/direct-shape.