# Fusing Hand and Body Skeletons for Human Action Recognition in Assembly

arXiv: https://arxiv.org/abs/2307.09238 · HTML: https://mlsanity.com/paper/2307.09238/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain](https://mlsanity.com/paper/2010.05654/) (2020)
- [PSUMNet: Unified Modality Part Streams are All You Need for Efficient Pose-based Action Recognition](https://mlsanity.com/paper/2208.05775/) (2022)
- [ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding](https://mlsanity.com/paper/2304.08210/) (2023)
- [NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions](https://mlsanity.com/paper/2101.11529/) (2021)
- [Efficient Multi-Task Scene Analysis with RGB-D Transformers](https://mlsanity.com/paper/2306.05242/) (2023)
- [How Object Information Improves Skeleton-based Human Action Recognition in Assembly Tasks](https://mlsanity.com/paper/2306.05844/) (2023)

## Built on by (most cited first, 1 of 1)

- [Including Semantic Information via Word Embeddings for Skeleton-based Action Recognition](https://mlsanity.com/paper/2506.18721/) (2025)

## Abstract (arXiv)

As collaborative robots (cobots) continue to gain popularity in industrial manufacturing, effective human-robot collaboration becomes crucial. Cobots should be able to recognize human actions to assist with assembly tasks and act autonomously. To achieve this, skeleton-based approaches are often used due to their ability to generalize across various people and environments. Although body skeleton approaches are widely used for action recognition, they may not be accurate enough for assembly actions where the worker's fingers and hands play a significant role. To address this limitation, we propose a method in which less detailed body skeletons are combined with highly detailed hand skeletons. We investigate CNNs and transformers, the latter of which are particularly adept at extracting and combining important information from both skeleton types using attention. This paper demonstrates the effectiveness of our proposed approach in enhancing action recognition in assembly scenarios.