# Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality

arXiv: https://arxiv.org/abs/2601.11535 · HTML: https://mlsanity.com/paper/2601.11535/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Builds on (most cited first, 7 of 7)

- [Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence](https://mlsanity.com/paper/2106.08710/) (2021)
- [Augmented Reality Applied to LEGO Construction: AR-based Building Instructions with High Accuracy & Precision and Realistic Object-Hand Occlusions](https://mlsanity.com/paper/1907.12549/) (2019)
- [Design Patterns for Situated Visualization in Augmented Reality](https://mlsanity.com/paper/2307.09157/) (2023)
- [Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly](https://mlsanity.com/paper/2409.18390/) (2024)
- [Making Physical Objects with Generative AI and Robotic Assembly: Considering Fabrication Constraints, Sustainability, Time, Functionality, and Accessibility](https://mlsanity.com/paper/2504.19131/) (2025)
- [Integrative Object and Pose to Task Detection for an Augmented-Reality-based Human Assistance System using Neural Networks](https://mlsanity.com/paper/2008.13419/) (2020)
- [AI Assisted AR Assembly: Object Recognition and Computer Vision for Augmented Reality Assisted Assembly](https://mlsanity.com/paper/2511.05394/) (2025)

## Abstract (arXiv)

Recent advances in augmented reality (AR) have enabled interactive systems that assist users in physical assembly tasks. In this paper, we present an AR-assisted assembly workflow that leverages object recognition and hand tracking to (1) identify custom components, (2) display step-by-step instructions, (3) detect assembly deviations, and (4) dynamically update the instructions based on users' hands-on interactions with physical parts. Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace. For each assembly step, it overlays bounding boxes in AR to indicate both the current position and the target placement of relevant components, while hand-tracking data verifies whether the user interacts with the correct part. Rather than enforcing a fixed sequence, the system highlights potential assembly errors and interprets user deviations as opportunities for iteration and creative exploration. A case study with LEGO blocks and custom 3D-printed components demonstrates how the system links digital instructions to physical assembly, eliminating the need for manual searching, sorting, or labeling of parts.