# Toward Ergonomic Risk Prediction via Segmentation of Indoor Object Manipulation Actions Using Spatiotemporal Convolutional Networks

arXiv: https://arxiv.org/abs/1902.05176 · HTML: https://mlsanity.com/paper/1902.05176/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2019.

## Built on by (most cited first, 3 of 3)

- [Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusion](https://mlsanity.com/paper/2309.06462/) (2023)
- [A vision-based framework for human behavior understanding in industrial assembly lines](https://mlsanity.com/paper/2409.17356/) (2024)
- [Data-Driven Ergonomic Risk Assessment of Complex Hand-intensive Manufacturing Processes](https://mlsanity.com/paper/2403.05591/) (2024)

## Abstract (arXiv)

Automated real-time prediction of the ergonomic risks of manipulating objects is a key unsolved challenge in developing effective human-robot collaboration systems for logistics and manufacturing applications. We present a foundational paradigm to address this challenge by formulating the problem as one of action segmentation from RGB-D camera videos. Spatial features are first learned using a deep convolutional model from the video frames, which are then fed sequentially to temporal convolutional networks to semantically segment the frames into a hierarchy of actions, which are either ergonomically safe, require monitoring, or need immediate attention. For performance evaluation, in addition to an open-source kitchen dataset, we collected a new dataset comprising twenty individuals picking up and placing objects of varying weights to and from cabinet and table locations at various heights. Results show very high (87-94)% F1 overlap scores among the ground truth and predicted frame labels for videos lasting over two minutes and consisting of a large number of actions.