# Evaluating multimodal emotion recognition in proactive conversational agents: A user study

arXiv: https://arxiv.org/abs/2605.20200 · HTML: https://mlsanity.com/paper/2605.20200/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 64 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 12 of 12)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2109.04919/) (2021)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations](https://mlsanity.com/paper/2306.01505/) (2023)
- [Generative Agent-Based Modeling: Unveiling Social System Dynamics through Coupling Mechanistic Models with Generative Artificial Intelligence](https://mlsanity.com/paper/2309.11456/) (2023)
- [A Survey on Facial Expression Recognition of Static and Dynamic Emotions](https://mlsanity.com/paper/2408.15777/) (2024)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [A Review on Text-Based Emotion Detection -- Techniques, Applications, Datasets, and Future Directions](https://mlsanity.com/paper/2205.03235/) (2022)
- [Affect Control Processes: Intelligent Affective Interaction using a Partially Observable Markov Decision Process](https://mlsanity.com/paper/1306.5279/) (2013)
- [Personality-affected Emotion Generation in Dialog Systems](https://mlsanity.com/paper/2404.07229/) (2024)
- ["Only ChatGPT gets me": An Empirical Analysis of GPT versus other Large Language Models for Emotion Detection in Text](https://mlsanity.com/paper/2503.04831/) (2025)
- [Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera](https://mlsanity.com/paper/2408.07982/) (2024)

## Abstract (arXiv)

This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence. The system evaluates real-time affective states through two distinct channels: a computer vision-based facial recognition module and a semantic linguistic analysis engine. To validate the framework, an empirical study was conducted with 20 users who engaged in dynamic, unscripted dialogues with the conversational agent. The findings reveal a significant discrepancy between automated visual cues and actual internal emotional states. When interacting with the AI, users consistently exhibited a "poker face" effect, displaying serious, concentrated facial expressions even when experiencing positive emotions. Consequently, the generative AI linguistic analysis proved significantly more reliable, by contextualizing the users' verbal expressions. Furthermore, an analysis of the interaction dynamics demonstrated that SIAs can effectively elicit specific emotions by adapting conversational themes and employing structured linguistic patterns, such as empathetic or humorous language. However, the study also noted that instances of uncalibrated proactivity occasionally led to user disengagement and a perception of artificiality. Ultimately, this research highlights the necessity of refining SIAs to dynamically adapt to users' emotional evolution, relying on deep linguistic context to foster more natural, human-like interactions.