# LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

arXiv: https://arxiv.org/abs/1707.04796 · HTML: https://mlsanity.com/paper/1707.04796/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2017.

## Built on by (most cited first, 30 of 32)

- [HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios](https://mlsanity.com/paper/2212.10428/) (2022)
- [PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects](https://mlsanity.com/paper/2205.08811/) (2022)
- [StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimation](https://mlsanity.com/paper/2109.10115/) (2021)
- [Towards Real-World Category-level Articulation Pose Estimation](https://mlsanity.com/paper/2105.03260/) (2021)
- [CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds](https://mlsanity.com/paper/2103.01977/) (2021)
- [TriVoC: Efficient Voting-based Consensus Maximization for Robust Point Cloud Registration with Extreme Outlier Ratios](https://mlsanity.com/paper/2111.00657/) (2021)
- [SALT: A Semi-automatic Labeling Tool for RGB-D Video Sequences](https://mlsanity.com/paper/2102.10820/) (2021)
- [Pseudo Flow Consistency for Self-Supervised 6D Object Pose Estimation](https://mlsanity.com/paper/2308.10016/) (2023)
- [Mobile Robot Manipulation using Pure Object Detection](https://mlsanity.com/paper/2201.12437/) (2022)
- [NARF22: Neural Articulated Radiance Fields for Configuration-Aware Rendering](https://mlsanity.com/paper/2210.01166/) (2022)
- [Semantic keypoint-based pose estimation from single RGB frames](https://mlsanity.com/paper/2204.05864/) (2022)
- [Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps](https://mlsanity.com/paper/2209.12744/) (2022)
- [ProgressLabeller: Visual Data Stream Annotation for Training Object-Centric 3D Perception](https://mlsanity.com/paper/2203.00283/) (2022)
- [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation](https://mlsanity.com/paper/2407.12449/) (2024)
- [DANIEL: A Fast and Robust Consensus Maximization Method for Point Cloud Registration with High Outlier Ratios](https://mlsanity.com/paper/2110.05075/) (2021)
- [Persistent Homology Meets Object Unity: Object Recognition in Clutter](https://mlsanity.com/paper/2305.03815/) (2023)
- [IFFNeRF: Initialisation Free and Fast 6DoF pose estimation from a single image and a NeRF model](https://mlsanity.com/paper/2403.12682/) (2024)
- [NeuV-SLAM: Fast Neural Multiresolution Voxel Optimization for RGBD Dense SLAM](https://mlsanity.com/paper/2402.02020/) (2024)
- [NeRF-Feat: 6D Object Pose Estimation using Feature Rendering](https://mlsanity.com/paper/2406.13796/) (2024)
- [Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimators](https://mlsanity.com/paper/2209.07976/) (2022)
- [Digital Twin Tracking Dataset (DTTD): A New RGB+Depth 3D Dataset for Longer-Range Object Tracking Applications](https://mlsanity.com/paper/2302.05991/) (2023)
- [Industrial Application of 6D Pose Estimation for Robotic Manipulation in Automotive Internal Logistics](https://mlsanity.com/paper/2309.14265/) (2023)
- [Jacquard V2: Refining Datasets using the Human In the Loop Data Correction Method](https://mlsanity.com/paper/2402.05747/) (2024)
- [PCR-99: A Practical Method for Point Cloud Registration with 99 Percent Outliers](https://mlsanity.com/paper/2402.16598/) (2024)
- [Semi-automatic 3D Object Keypoint Annotation and Detection for the Masses](https://mlsanity.com/paper/2201.07665/) (2022)
- [RF-Annotate: Automatic RF-Supervised Image Annotation of Common Objects in Context](https://mlsanity.com/paper/2211.08837/) (2022)
- [Robust 6DoF Pose Estimation Against Depth Noise and a Comprehensive Evaluation on a Mobile Dataset](https://mlsanity.com/paper/2309.13570/) (2023)
- [SPLATART: Articulated Gaussian Splatting with Estimated Object Structure](https://mlsanity.com/paper/2506.12184/) (2025)
- [3D Annotation Of Arbitrary Objects In The Wild](https://mlsanity.com/paper/2109.07165/) (2021)
- [TAILOR: Teaching with Active and Incremental Learning for Object Registration](https://mlsanity.com/paper/2205.11692/) (2022)

## Abstract (arXiv)

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.