# Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer

arXiv: https://arxiv.org/abs/1511.03240 · HTML: https://mlsanity.com/paper/1511.03240/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2015.

## Built on by (most cited first, 30 of 39)

- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAM](https://mlsanity.com/paper/2103.05056/) (2021)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types](https://mlsanity.com/paper/2103.13318/) (2021)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [Text2Pos: Text-to-Point-Cloud Cross-Modal Localization](https://mlsanity.com/paper/2203.15125/) (2022)
- [CT-ICP: Real-time Elastic LiDAR Odometry with Loop Closure](https://mlsanity.com/paper/2109.12979/) (2021)
- [Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.11062/) (2021)
- [Domain generalization of 3D semantic segmentation in autonomous driving](https://mlsanity.com/paper/2212.04245/) (2022)
- [Exploring Event-driven Dynamic Context for Accident Scene Segmentation](https://mlsanity.com/paper/2112.05006/) (2021)
- [CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data](https://mlsanity.com/paper/2310.18773/) (2023)
- [Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centers](https://mlsanity.com/paper/2106.12442/) (2021)
- [Asynchronous Multi-View SLAM](https://mlsanity.com/paper/2101.06562/) (2021)
- [Click to Move: Controlling Video Generation with Sparse Motion](https://mlsanity.com/paper/2108.08815/) (2021)
- [Ad-datasets: a meta-collection of data sets for autonomous driving](https://mlsanity.com/paper/2202.01909/) (2022)
- [SALT: A Semi-automatic Labeling Tool for RGB-D Video Sequences](https://mlsanity.com/paper/2102.10820/) (2021)
- [PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes](https://mlsanity.com/paper/2309.10815/) (2023)
- [Recalibrating the KITTI Dataset Camera Setup for Improved Odometry Accuracy](https://mlsanity.com/paper/2109.03462/) (2021)
- [SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator](https://mlsanity.com/paper/2109.10270/) (2021)
- [HPointLoc: Point-based Indoor Place Recognition using Synthetic RGB-D Images](https://mlsanity.com/paper/2212.14649/) (2022)
- [UrbanNet: Leveraging Urban Maps for Long Range 3D Object Detection](https://mlsanity.com/paper/2110.05561/) (2021)
- [Multi-modal NeRF Self-Supervision for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2411.02969/) (2024)
- [COLA: COarse LAbel pre-training for 3D semantic segmentation of sparse LiDAR datasets](https://mlsanity.com/paper/2202.06884/) (2022)
- [Vision-based Large-scale 3D Semantic Mapping for Autonomous Driving Applications](https://mlsanity.com/paper/2203.01087/) (2022)
- [Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics](https://mlsanity.com/paper/2303.04302/) (2023)
- [Fluorescent Neuronal Cells v2: Multi-Task, Multi-Format Annotations for Deep Learning in Microscopy](https://mlsanity.com/paper/2307.14243/) (2023)
- [Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression](https://mlsanity.com/paper/2107.04497/) (2021)

## Abstract (arXiv)

Semantic annotations are vital for training models for object recognition, semantic segmentation or scene understanding. Unfortunately, pixelwise annotation of images at very large scale is labor-intensive and only little labeled data is available, particularly at instance level and for street scenes. In this paper, we propose to tackle this problem by lifting the semantic instance labeling task from 2D into 3D. Given reconstructions from stereo or laser data, we annotate static 3D scene elements with rough bounding primitives and develop a model which transfers this information into the image domain. We leverage our method to obtain 2D labels for a novel suburban video dataset which we have collected, resulting in 400k semantic and instance image annotations. A comparison of our method to state-of-the-art label transfer baselines reveals that 3D information enables more efficient annotation while at the same time resulting in improved accuracy and time-coherent labels.