# ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving

arXiv: https://arxiv.org/abs/1811.12222 · HTML: https://mlsanity.com/paper/1811.12222/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2018.

## Built on by (most cited first, 30 of 59)

- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset](https://mlsanity.com/paper/2107.04286/) (2021)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection](https://mlsanity.com/paper/2108.11127/) (2021)
- [Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task](https://mlsanity.com/paper/2203.13608/) (2022)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [Going Denser with Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2305.11173/) (2023)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Pose for Everything: Towards Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2207.10387/) (2022)
- [Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset](https://mlsanity.com/paper/2309.13549/) (2023)
- [ACR-Pose: Adversarial Canonical Representation Reconstruction Network for Category Level 6D Object Pose Estimation](https://mlsanity.com/paper/2111.10524/) (2021)
- [Graph Neural Network and Spatiotemporal Transformer Attention for 3D Video Object Detection from Point Clouds](https://mlsanity.com/paper/2207.12659/) (2022)
- [A Graph-Based Approach for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2311.17891/) (2023)
- [Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles](https://mlsanity.com/paper/2103.14098/) (2021)
- [IAFA: Instance-aware Feature Aggregation for 3D Object Detection from a Single Image](https://mlsanity.com/paper/2103.03480/) (2021)
- [Optimal Pose and Shape Estimation for Category-level 3D Object Perception](https://mlsanity.com/paper/2104.08383/) (2021)
- [Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection](https://mlsanity.com/paper/2212.05265/) (2022)
- [Deep Transfer Learning for Intelligent Vehicle Perception: a Survey](https://mlsanity.com/paper/2306.15110/) (2023)
- [CapeX: Category-Agnostic Pose Estimation from Textual Point Explanation](https://mlsanity.com/paper/2406.00384/) (2024)
- [Keypoint Communities](https://mlsanity.com/paper/2110.00988/) (2021)
- [Reducing Overconfidence Predictions for Autonomous Driving Perception](https://mlsanity.com/paper/2202.07825/) (2022)
- [Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints](https://mlsanity.com/paper/2206.12498/) (2022)
- [Deeper into Self-Supervised Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2312.01283/) (2023)
- [Meta-Point Learning and Refining for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2403.13647/) (2024)
- [PartImageNet++ Dataset: Scaling up Part-based Models for Robust Recognition](https://mlsanity.com/paper/2407.10918/) (2024)
- [RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image](https://mlsanity.com/paper/2410.08181/) (2024)
- [Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild](https://mlsanity.com/paper/2111.12728/) (2021)

## Abstract (arXiv)

Autonomous driving has attracted remarkable attention from both industry and academia. An important task is to estimate 3D properties(e.g.translation, rotation and shape) of a moving or parked vehicle on the road. This task, while critical, is still under-researched in the computer vision community - partially owing to the lack of large scale and fully-annotated 3D car database suitable for autonomous driving research. In this paper, we contribute the first large-scale database suitable for 3D car instance understanding - ApolloCar3D. The dataset contains 5,277 driving images and over 60K car instances, where each car is fitted with an industry-grade 3D CAD model with absolute model size and semantically labelled keypoints. This dataset is above 20 times larger than PASCAL3D+ and KITTI, the current state-of-the-art. To enable efficient labelling in 3D, we build a pipeline by considering 2D-3D keypoint correspondences for a single instance and 3D relationship among multiple instances. Equipped with such dataset, we build various baseline algorithms with the state-of-the-art deep convolutional neural networks. Specifically, we first segment each car with a pre-trained Mask R-CNN, and then regress towards its 3D pose and shape based on a deformable 3D car model with or without using semantic keypoints. We show that using keypoints significantly improves fitting performance. Finally, we develop a new 3D metric jointly considering 3D pose and 3D shape, allowing for comprehensive evaluation and ablation study. By comparing with human performance we suggest several future directions for further improvements.