# Augmented Reality Applied to LEGO Construction: AR-based Building Instructions with High Accuracy & Precision and Realistic Object-Hand Occlusions

arXiv: https://arxiv.org/abs/1907.12549 · HTML: https://mlsanity.com/paper/1907.12549/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2019.

## Built on by (most cited first, 6 of 6)

- [Thermodynamics-informed neural networks for physically realistic mixed reality](https://mlsanity.com/paper/2210.13414/) (2022)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions for Assembly](https://mlsanity.com/paper/2311.16337/) (2023)
- [BrickPal: Augmented Reality-based Assembly Instructions for Brick Models](https://mlsanity.com/paper/2307.03162/) (2023)
- [EVER: Edge-Assisted Auto-Verification for Mobile MR-Aided Operation](https://mlsanity.com/paper/2510.18224/) (2025)
- [Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality](https://mlsanity.com/paper/2601.11535/) (2025)

## Abstract (arXiv)

BRICKxAR is a novel Augmented Reality (AR) instruction method for construction toys such as LEGO. With BRICKxAR, physical LEGO construction is guided by virtual bricks. Compared with the state-of-the-art, accuracy of the virtual - physical model alignment is significantly improved through a new design of marker-based registration, which can achieve an average error less than 1mm throughout the model. Realistic object occlusion is accomplished to reveal the true spatial relationship between physical and virtual bricks. LEGO players' hand detection and occlusion are realized to visualize the correct spatial relationship between real hands and virtual bricks, and allow virtual bricks to be "grasped" by real hands. The integration of these features makes AR instructions possible for small-parts assembly, validated through a working AR prototype for constructing LEGO Arc de Triomphe, quantitative measures of the accuracies of registration and occlusions, and heuristic evaluation of AR instruction features.