# Real-Time Human-Robot Interaction Intent Detection Using RGB-based Pose and Emotion Cues with Cross-Camera Model Generalization

arXiv: https://arxiv.org/abs/2512.17958 · HTML: https://mlsanity.com/paper/2512.17958/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [An Alternative Probabilistic Interpretation of the Huber Loss](https://mlsanity.com/paper/1911.02088/) (2019)
- [3D Human Pose Estimation in RGBD Images for Robotic Task Learning](https://mlsanity.com/paper/1803.02622/) (2018)
- [Interact as You Intend: Intention-Driven Human-Object Interaction Detection](https://mlsanity.com/paper/1808.09796/) (2018)
- [Towards Generating Real-World Time Series Data](https://mlsanity.com/paper/2111.08386/) (2021)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Intention estimation from gaze and motion features for human-robot shared-control object manipulation](https://mlsanity.com/paper/2208.08688/) (2022)
- [InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions](https://mlsanity.com/paper/2311.12943/) (2023)
- [Model Predictive Control for Fluid Human-to-Robot Handovers](https://mlsanity.com/paper/2204.00134/) (2022)
- [Sensor Data Augmentation from Skeleton Pose Sequences for Improving Human Activity Recognition](https://mlsanity.com/paper/2406.16886/) (2024)
- [Self-Supervised Prediction of the Intention to Interact with a Service Robot](https://mlsanity.com/paper/2309.07477/) (2023)
- [Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction](https://mlsanity.com/paper/2402.04768/) (2024)
- [Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation](https://mlsanity.com/paper/2401.06563/) (2024)
- [Predicting the Intention to Interact with a Service Robot:the Role of Gaze Cues](https://mlsanity.com/paper/2404.01986/) (2024)
- [Gaze-Based Intention Recognition for Human-Robot Collaboration](https://mlsanity.com/paper/2405.07570/) (2024)
- [PIM: Physics-Informed Multi-task Pre-training for Improving Inertial Sensor-Based Human Activity Recognition](https://mlsanity.com/paper/2503.17978/) (2025)

## Built on by (most cited first, 1 of 1)

- [Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset](https://mlsanity.com/paper/2606.03694/) (2026)

## Abstract (arXiv)

Service robots in public spaces require real-time understanding of human behavioral intentions for natural interaction. We present a practical multimodal framework for frame-accurate human-robot interaction intent detection that fuses camera-invariant 2D skeletal pose and facial emotion features extracted from monocular RGB video. Unlike prior methods requiring RGB-D sensors or GPU acceleration, our approach resource-constrained embedded hardware (Raspberry Pi 5, CPU-only). To address the severe class imbalance in natural human-robot interaction datasets, we introduce a novel approach to synthesize temporally coherent pose-emotion-label sequences for data re-balancing called MINT-RVAE (Multimodal Recurrent Variational Autoencoder for Intent Sequence Generation). Comprehensive offline evaluations under cross-subject and cross-scene protocols demonstrate strong generalization performance, achieving frame- and sequence-level AUROC of 0.95. Crucially, we validate real-world generalization through cross-camera evaluation on the MIRA robot head, which employs a different onboard RGB sensor and operates in uncontrolled environments not represented in the training data. Despite this domain shift, the deployed system achieves 91% accuracy and 100% recall across 32 live interaction trials. The close correspondence between offline and deployed performance confirms the cross-sensor and cross-environment robustness of the proposed multimodal approach, highlighting its suitability for ubiquitous multimedia-enabled social robots.