# Hierarchical Deep Learning for Intention Estimation of Teleoperation Manipulation in Assembly Tasks

arXiv: https://arxiv.org/abs/2403.19770 · HTML: https://mlsanity.com/paper/2403.19770/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Waypoint-Based Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2307.14326/) (2023)
- [Motion Planning around Obstacles with Convex Optimization](https://mlsanity.com/paper/2205.04422/) (2022)
- [Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction](https://mlsanity.com/paper/1802.10503/) (2018)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [Deep Hierarchical Classification for Category Prediction in E-commerce System](https://mlsanity.com/paper/2005.06692/) (2020)
- [Hierarchical Intention Tracking for Robust Human-Robot Collaboration in Industrial Assembly Tasks](https://mlsanity.com/paper/2203.09063/) (2022)

## Built on by (most cited first, 1 of 1)

- [STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration](https://mlsanity.com/paper/2608.27225/) (2026)

## Abstract (arXiv)

In human-robot collaboration, shared control presents an opportunity to teleoperate robotic manipulation to improve the efficiency of manufacturing and assembly processes. Robots are expected to assist in executing the user's intentions. To this end, robust and prompt intention estimation is needed, relying on behavioral observations. The framework presents an intention estimation technique at hierarchical levels i.e., low-level actions and high-level tasks, by incorporating multi-scale hierarchical information in neural networks. Technically, we employ hierarchical dependency loss to boost overall accuracy. Furthermore, we propose a multi-window method that assigns proper hierarchical prediction windows of input data. An analysis of the predictive power with various inputs demonstrates the predominance of the deep hierarchical model in the sense of prediction accuracy and early intention identification. We implement the algorithm on a virtual reality (VR) setup to teleoperate robotic hands in a simulation with various assembly tasks to show the effectiveness of online estimation.