# Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

arXiv: https://arxiv.org/abs/2312.05265 · HTML: https://mlsanity.com/paper/2312.05265/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Counting Out Time: Class Agnostic Video Repetition Counting in the Wild](https://mlsanity.com/paper/2006.15418/) (2020)
- [LAEO-Net: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/1906.05261/) (2019)
- [Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases](https://mlsanity.com/paper/1909.12911/) (2019)
- [An Attention Model for group-level emotion recognition](https://mlsanity.com/paper/1807.03380/) (2018)
- [Emotion Recognition in the Wild using Deep Neural Networks and Bayesian Classifiers](https://mlsanity.com/paper/1709.03820/) (2017)
- [Group-Level Emotion Recognition Using a Unimodal Privacy-Safe Non-Individual Approach](https://mlsanity.com/paper/2009.07013/) (2020)
- [Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention](https://mlsanity.com/paper/2111.05890/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Survey of Deep Learning for Group-level Emotion Recognition](https://mlsanity.com/paper/2408.15276/) (2024)
- [Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition](https://mlsanity.com/paper/2604.02397/) (2026)
- [GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos](https://mlsanity.com/paper/2604.16214/) (2026)

## Abstract (arXiv)

This paper explores privacy-compliant group-level emotion recognition ''in-the-wild'' within the EmotiW Challenge 2023. Group-level emotion recognition can be useful in many fields including social robotics, conversational agents, e-coaching and learning analytics. This research imposes itself using only global features avoiding individual ones, i.e. all features that can be used to identify or track people in videos (facial landmarks, body poses, audio diarization, etc.). The proposed multimodal model is composed of a video and an audio branches with a cross-attention between modalities. The video branch is based on a fine-tuned ViT architecture. The audio branch extracts Mel-spectrograms and feed them through CNN blocks into a transformer encoder. Our training paradigm includes a generated synthetic dataset to increase the sensitivity of our model on facial expression within the image in a data-driven way. The extensive experiments show the significance of our methodology. Our privacy-compliant proposal performs fairly on the EmotiW challenge, with 79.24% and 75.13% of accuracy respectively on validation and test set for the best models. Noticeably, our findings highlight that it is possible to reach this accuracy level with privacy-compliant features using only 5 frames uniformly distributed on the video.