# MINT-RVAE: Multi-Cues Intention Prediction of Human-Robot Interaction using Human Pose and Emotion Information from RGB-only Camera Data

arXiv: https://arxiv.org/abs/2509.22573 · HTML: https://mlsanity.com/paper/2509.22573/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [An Alternative Probabilistic Interpretation of the Huber Loss](https://mlsanity.com/paper/1911.02088/) (2019)
- [Towards Generating Real-World Time Series Data](https://mlsanity.com/paper/2111.08386/) (2021)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions](https://mlsanity.com/paper/2311.12943/) (2023)
- [Model Predictive Control for Fluid Human-to-Robot Handovers](https://mlsanity.com/paper/2204.00134/) (2022)
- [Risk-Calibrated Human-Robot Interaction via Set-Valued Intent Prediction](https://mlsanity.com/paper/2403.15959/) (2024)
- [Self-Supervised Prediction of the Intention to Interact with a Service Robot](https://mlsanity.com/paper/2309.07477/) (2023)
- [Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction](https://mlsanity.com/paper/2402.04768/) (2024)
- [Predicting the Intention to Interact with a Service Robot:the Role of Gaze Cues](https://mlsanity.com/paper/2404.01986/) (2024)
- [Bayesian Intention for Enhanced Human Robot Collaboration](https://mlsanity.com/paper/2410.00302/) (2024)

## Built on by (most cited first, 1 of 1)

- [HUI360: A 360deg Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation](https://mlsanity.com/paper/2608.11051/) (2026)

## Abstract (arXiv)

Efficiently detecting human intent to interact with ubiquitous robots is crucial for effective human-robot interaction (HRI) and collaboration. Over the past decade, deep learning has gained traction in this field, with most existing approaches relying on multimodal inputs, such as RGB combined with depth (RGB-D), to classify time-sequence windows of sensory data as interactive or non-interactive. In contrast, we propose a novel RGB-only pipeline for predicting human interaction intent with frame-level precision, enabling faster robot responses and improved service quality. A key challenge in intent prediction is the class imbalance inherent in real-world HRI datasets, which can hinder the model's training and generalization. To address this, we introduce MINT-RVAE, a synthetic sequence generation method, along with new loss functions and training strategies that enhance generalization on out-of-sample data. Our approach achieves state-of-the-art performance (AUROC: 0.95) outperforming prior works (AUROC: 0.90-0.912), while requiring only RGB input and supporting precise frame onset prediction. Finally, to support future research, we openly release our new dataset with frame-level labeling of human interaction intent.