mlsanityData to 14 Oct 2026

Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coach

arXiv 2311.05567 ↗First posted 9 Nov 2023cs.CV cs.HC cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The EMPATHIC project aimed to design an emotionally expressive virtual coach capable of engaging healthy seniors to improve well-being and promote independent aging. One of the core aspects of the system is its human sensing capabilities, allowing for the perception of emotional states to provide a personalized experience. This paper outlines the development of the emotion expression recognition module of the virtual coach, encompassing data collection, annotation design, and a first methodological approach, all tailored to the project requirements. With the latter, we investigate the role of various modalities, individually and combined, for discrete emotion expression recognition in this context: speech from audio, and facial expressions, gaze, and head dynamics from video. The collected corpus includes users from Spain, France, and Norway, and was annotated separately for the audio and video channels with distinct emotional labels, allowing for a performance comparison across cultures and label types. Results confirm the informative power of the modalities studied for the emotional categories considered, with multimodal methods generally outperforming others (around 68% accuracy with audio labels and 72-74% with video labels). The findings are expected to contribute to the limited literature on emotion recognition applied to older adults in conversational human-machine interaction.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 122 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

27 of 122 references.

References · 27

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  7. 2018Deep Facial Expression Recognition: A Surveycited by 270
  8. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  9. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  10. 2020ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variationcited by 141
  11. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  12. 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 70
15 more
  1. 2019SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wildcited by 62
  2. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  3. 2016Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applicationscited by 45
  4. 2017FaceBoxes: A CPU Real-time Face Detector with High Accuracycited by 45
  5. 2015Going Deeper in Facial Expression Recognition using Deep Neural Networkscited by 43
  6. 2021Automatic Gaze Analysis: A Survey of Deep Learning based Approachescited by 33
  7. 2019Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applicationscited by 29
  8. 2019Deep Learning for Human Affect Recognition: Insights and New Developmentscited by 23
  9. 2019Speech Emotion Recognition with Dual-Sequence LSTM Architecturecited by 23
  10. 2022Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?cited by 16
  11. 2022Gaze-enhanced Crossmodal Embeddings for Emotion Recognitioncited by 4
  12. 2018Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect predictioncited by 2
  13. 2019Eye-based Continuous Affect Predictioncited by 2
  14. 2021The EMPATHIC Project: Mid-term Achievementscited by 2
  15. 2019Speech, Head, and Eye-based Cues for Continuous Affect Predictioncited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Multi-face emotion detection for effective Human-Robot Interactioncited by 1
  2. 2025Experimenting with Affective Computing Models in Video Interviews with Spanish-speaking Older Adultscited by 0
  3. 2025Towards Affect-Adaptive Human-Robot Interaction: A Protocol for Multimodal Dataset Collection on Social Anxietycited by 0