# You Only Label Once: 3D Box Adaptation from Point Cloud to Image via Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2211.09302 · HTML: https://mlsanity.com/paper/2211.09302/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [HDMapNet: An Online HD Map Construction and Evaluation Framework](https://mlsanity.com/paper/2107.06307/) (2021)
- [VectorMapNet: End-to-end Vectorized HD Map Learning](https://mlsanity.com/paper/2206.08920/) (2022)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [Stereo R-CNN based 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09738/) (2019)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors](https://mlsanity.com/paper/1911.11288/) (2019)
- [Weakly Supervised 3D Object Detection from Lidar Point Cloud](https://mlsanity.com/paper/2007.11901/) (2020)
- [Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving](https://mlsanity.com/paper/2210.08061/) (2022)
- [Auto4D: Learning to Label 4D Objects from Sequential Point Clouds](https://mlsanity.com/paper/2101.06586/) (2021)
- [RoadMap: A Light-Weight Semantic Map for Visual Localization towards Autonomous Driving](https://mlsanity.com/paper/2106.02527/) (2021)
- [ILabel: Interactive Neural Scene Labelling](https://mlsanity.com/paper/2111.14637/) (2021)
- [Stereo Vision-based Semantic 3D Object and Ego-motion Tracking for Autonomous Driving](https://mlsanity.com/paper/1807.02062/) (2018)
- [Unsupervised Object Detection with LiDAR Clues](https://mlsanity.com/paper/2011.12953/) (2020)
- [Semantic Segmentation of 3D LiDAR Data in Dynamic Scene Using Semi-supervised Learning](https://mlsanity.com/paper/1809.00426/) (2018)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [Leveraging Pre-Trained 3D Object Detection Models For Fast Ground Truth Generation](https://mlsanity.com/paper/1807.06072/) (2018)
- [LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images](https://mlsanity.com/paper/1910.13955/) (2019)
- [Automatic labelling of urban point clouds using data fusion](https://mlsanity.com/paper/2108.13757/) (2021)
- [Coarse-To-Fine Visual Localization Using Semantic Compact Map](https://mlsanity.com/paper/1910.04936/) (2019)

## Built on by (most cited first, 1 of 1)

- [CAMOT: Camera Angle-aware Multi-Object Tracking](https://mlsanity.com/paper/2409.17533/) (2024)

## Abstract (arXiv)

The image-based 3D object detection task expects that the predicted 3D bounding box has a ``tightness'' projection (also referred to as cuboid), which fits the object contour well on the image while still keeping the geometric attribute on the 3D space, e.g., physical dimension, pairwise orthogonal, etc. These requirements bring significant challenges to the annotation. Simply projecting the Lidar-labeled 3D boxes to the image leads to non-trivial misalignment, while directly drawing a cuboid on the image cannot access the original 3D information. In this work, we propose a learning-based 3D box adaptation approach that automatically adjusts minimum parameters of the 360circ Lidar 3D bounding box to perfectly fit the image appearance of panoramic cameras. With only a few 2D boxes annotation as guidance during the training phase, our network can produce accurate image-level cuboid annotations with 3D properties from Lidar boxes. We call our method ``you only label once'', which means labeling on the point cloud once and automatically adapting to all surrounding cameras. As far as we know, we are the first to focus on image-level cuboid refinement, which balances the accuracy and efficiency well and dramatically reduces the labeling effort for accurate cuboid annotation. Extensive experiments on the public Waymo and NuScenes datasets show that our method can produce human-level cuboid annotation on the image without needing manual adjustment.