# A Synthetic-Driven Vision System for Assembly Step Recognition

arXiv: https://arxiv.org/abs/2607.00129 · HTML: https://mlsanity.com/paper/2607.00129/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [FoundationStereo: Zero-Shot Stereo Matching](https://mlsanity.com/paper/2501.09898/) (2025)
- [TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding](https://mlsanity.com/paper/2401.08399/) (2024)
- [OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion](https://mlsanity.com/paper/2403.19417/) (2024)
- [GraspXL: Generating Grasping Motions for Diverse Objects at Scale](https://mlsanity.com/paper/2403.19649/) (2024)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesis](https://mlsanity.com/paper/2303.15469/) (2023)
- [ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation](https://mlsanity.com/paper/2309.03891/) (2023)
- [GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities](https://mlsanity.com/paper/2412.04244/) (2024)
- [BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects](https://mlsanity.com/paper/2412.05066/) (2024)
- [RobustDexGrasp: Robust Dexterous Grasping of General Objects](https://mlsanity.com/paper/2504.05287/) (2025)
- [On the Utility of 3D Hand Poses for Action Recognition](https://mlsanity.com/paper/2403.09805/) (2024)
- [ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding](https://mlsanity.com/paper/2304.08210/) (2023)
- [POV-Surgery: A Dataset for Egocentric Hand and Tool Pose Estimation During Surgical Activities](https://mlsanity.com/paper/2307.10387/) (2023)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [Deep Learning Models for Visual Inspection on Automotive Assembling Line](https://mlsanity.com/paper/2007.01857/) (2020)
- [Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations](https://mlsanity.com/paper/2512.19583/) (2025)
- [Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environment](https://mlsanity.com/paper/2311.11039/) (2023)

## Abstract (arXiv)

Quality control in industrial assembly is essential, and real-time monitoring of the assembly process is crucial for preventing costly defects and ensuring production reliability. Vision-based automated inspection offers a powerful solution for such real-time monitoring. However, due to the specialized industrial components and processes, training these models typically relies on task-specific real-world data, which is costly and labor-intensive to collect and annotate. In this paper, we propose a system that automatically generates realistic assembly sequences and further trains real-time inspection models using the synthetic data. It can be efficiently applied to a given task within an hour, requiring only CAD models and simple step descriptions. Focusing on practical challenges, our system integrates a physics-based motion generation module to capture the variance of different human assembly, designs domain-randomized rendering to deal with the environmental complexity and variation, and employs an object-detection-based step recognition module for robust sim-to-real transfer, leading to 92.4% accuracy on a real-world assembly case with 46.7%, 15.8% and 61.2% performance improvement, respectively. Overall, our system provides a practical solution for industrial assembly inspection without requiring expensive real-world data collection and annotation, with the effectiveness validated on real industrial assembly tasks.