# User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals

arXiv: https://arxiv.org/abs/2507.23544 · HTML: https://mlsanity.com/paper/2507.23544/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Builds on (most cited first, 10 of 10)

- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Expression Snippet Transformer for Robust Video-based Facial Expression Recognition](https://mlsanity.com/paper/2109.08409/) (2021)
- [LOGO-Former: Local-Global Spatio-Temporal Transformer for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2305.03343/) (2023)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions](https://mlsanity.com/paper/2203.10518/) (2022)
- [Behavioral assessment of a humanoid robot when attracting pedestrians in a mall](https://mlsanity.com/paper/2109.02771/) (2021)
- [FSER: Deep Convolutional Neural Networks for Speech Emotion Recognition](https://mlsanity.com/paper/2109.07916/) (2021)
- [Service Robots in a Bakery Shop: A Field Study](https://mlsanity.com/paper/2208.09260/) (2022)

## Abstract (arXiv)

In recent years, the demand for social robots has grown, requiring them to adapt their behaviors based on users' states. Accurately assessing user experience (UX) in human-robot interaction (HRI) is crucial for achieving this adaptability. UX is a multi-faceted measure encompassing aspects such as sentiment and engagement, yet existing methods often focus on these individually. This study proposes a UX estimation method for HRI by leveraging multimodal social signals. We construct a UX dataset and develop a Transformer-based model that utilizes facial expressions and voice for estimation. Unlike conventional models that rely on momentary observations, our approach captures both short- and long-term interaction patterns using a multi-instance learning framework. This enables the model to capture temporal dynamics in UX, providing a more holistic representation. Experimental results demonstrate that our method outperforms third-party human evaluators in UX estimation.