# ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation arXiv: https://arxiv.org/abs/2403.16400 ยท HTML: https://mlsanity.com/paper/2403.16400/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 63 references that are in the map. - First posted on arXiv 25 Mar 2024. ## Builds on (most cited first, 30 of 30) - [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015) - [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017) - [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017) - [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019) - [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018) - [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021) - [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018) - [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018) - [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019) - [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017) - [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017) - [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020) - [GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimator](https://mlsanity.com/paper/2102.12145/) (2021) - [FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation](https://mlsanity.com/paper/2103.02242/) (2021) - [HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects](https://mlsanity.com/paper/1904.03167/) (2019) - [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023) - [PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects](https://mlsanity.com/paper/2205.08811/) (2022) - [Unity Perception: Generate Synthetic Data for Computer Vision](https://mlsanity.com/paper/2107.04259/) (2021) - [UnrealCV: Connecting Computer Vision to Unreal Engine](https://mlsanity.com/paper/1609.01326/) (2016) - [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022) - [IKEA-Manual: Seeing Shape Assembly Step by Step](https://mlsanity.com/paper/2302.01881/) (2023) - [SRT3D: A Sparse Region-Based 3D Object Tracking Approach for the Real World](https://mlsanity.com/paper/2110.12715/) (2021) - [YOLOPose: Transformer-based Multi-Object 6D Pose Estimation using Keypoint Regression](https://mlsanity.com/paper/2205.02536/) (2022) - [Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer](https://mlsanity.com/paper/2011.07112/) (2020) - [PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation](https://mlsanity.com/paper/2211.14125/) (2022) - [Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking](https://mlsanity.com/paper/2302.11458/) (2023) - [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024) - [IKEA Object State Dataset: A 6DoF object pose estimation dataset and benchmark for multi-state assembly objects](https://mlsanity.com/paper/2111.08614/) (2021) - [A Multi-body Tracking Framework - From Rigid Objects to Kinematic Structures](https://mlsanity.com/paper/2208.01502/) (2022) - [For A More Comprehensive Evaluation of 6DoF Object Pose Tracking](https://mlsanity.com/paper/2309.07796/) (2023) ## Built on by (most cited first, 6 of 6) - [Supervised Representation Learning towards Generalizable Assembly State Recognition](https://mlsanity.com/paper/2408.11700/) (2024) - [Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling](https://mlsanity.com/paper/2510.12385/) (2025) - [MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects](https://mlsanity.com/paper/2603.22839/) (2026) - [Find the Assembly Mistakes: Error Segmentation for Industrial Applications](https://mlsanity.com/paper/2408.12945/) (2024) - [Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency](https://mlsanity.com/paper/2606.04992/) (2026) - [A Synthetic-Driven Vision System for Assembly Step Recognition](https://mlsanity.com/paper/2607.00129/) (2026) ## Abstract (arXiv) In medical and industrial domains, providing guidance for assembly processes can be critical to ensure efficiency and safety. Errors in assembly can lead to significant consequences such as extended surgery times and prolonged manufacturing or maintenance times in industry. Assembly scenarios can benefit from in-situ augmented reality visualization, i.e., augmentations in close proximity to the target object, to provide guidance, reduce assembly times, and minimize errors. In order to enable in-situ visualization, 6D pose estimation can be leveraged to identify the correct location for an augmentation. Existing 6D pose estimation techniques primarily focus on individual objects and static captures. However, assembly scenarios have various dynamics, including occlusion during assembly and dynamics in the appearance of assembly objects. Existing work focus either on object detection combined with state detection, or focus purely on the pose estimation. To address the challenges of 6D pose estimation in combination with assembly state detection, our approach ASDF builds upon the strengths of YOLOv8, a real-time capable object detection framework. We extend this framework, refine the object pose, and fuse pose knowledge with network-detected pose information. Utilizing our late fusion in our Pose2State module results in refined 6D pose estimation and assembly state detection. By combining both pose and state information, our Pose2State module predicts the final assembly state with precision. The evaluation of our ASDF dataset shows that our Pose2State module leads to an improved assembly state detection and that the improvement of the assembly state further leads to a more robust 6D pose estimation. Moreover, on the GBOT dataset, we outperform the pure deep learning-based network and even outperform the hybrid and pure tracking-based approaches.