# HUI360: A 360deg Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

arXiv: https://arxiv.org/abs/2608.11051 · HTML: https://mlsanity.com/paper/2608.11051/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy](https://mlsanity.com/paper/1802.01186/) (2018)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Self-Supervised Prediction of the Intention to Interact with a Service Robot](https://mlsanity.com/paper/2309.07477/) (2023)
- [Towards engagement models that consider individual factors in HRI: on the relation of extroversion and negative attitude towards robots to gaze and speech during a human-robot assembly task](https://mlsanity.com/paper/1508.04603/) (2015)
- [On-the-fly Detection of User Engagement Decrease in Spontaneous Human-Robot Interaction, International Journal of Social Robotics, 2019](https://mlsanity.com/paper/2004.09596/) (2020)
- [Starting engagement detection towards a companion robot using multimodal features](https://mlsanity.com/paper/1503.03732/) (2015)
- [Joint Engagement Classification using Video Augmentation Techniques for Multi-person Human-robot Interaction](https://mlsanity.com/paper/2212.14128/) (2022)
- [MINT-RVAE: Multi-Cues Intention Prediction of Human-Robot Interaction using Human Pose and Emotion Information from RGB-only Camera Data](https://mlsanity.com/paper/2509.22573/) (2025)

## Built on by (most cited first, 1 of 1)

- [Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities](https://mlsanity.com/paper/2609.07394/) (2026)

## Abstract (arXiv)

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.