# Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach

arXiv: https://arxiv.org/abs/2606.07585 · HTML: https://mlsanity.com/paper/2606.07585/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 262 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis](https://mlsanity.com/paper/2501.09502/) (2025)
- [NPU-NTU System for Voice Privacy 2024 Challenge](https://mlsanity.com/paper/2409.04173/) (2024)
- [AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition](https://mlsanity.com/paper/2407.07653/) (2024)
- [A vector quantized masked autoencoder for audiovisual speech emotion recognition](https://mlsanity.com/paper/2305.03568/) (2023)
- [Synthetic Data: Revisiting the Privacy-Utility Trade-off](https://mlsanity.com/paper/2407.07926/) (2024)
- [Adversarial Auto-encoders for Speech Based Emotion Recognition](https://mlsanity.com/paper/1806.02146/) (2018)
- [DELTA: Variational Disentangled Learning for Privacy-Preserving Data Reprogramming](https://mlsanity.com/paper/2509.00693/) (2025)
- [Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention](https://mlsanity.com/paper/2111.05890/) (2021)

## Abstract (arXiv)

This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation. Unlike traditional emotion recognition methods that rely on individual-level cues such as face, gaze, or voice analysis, this work uses collective audio-video signals to infer emotions at the group level, reducing risks of individual monitoring and surveillance. Two complementary frameworks are proposed. The first is a cross-attention multimodal architecture for audio-video fusion, combined with Frames Attention Pooling (FAP) for temporal aggregation. It is supported by synthetic data augmentation and validated through ablation studies, demonstrating robustness in real-world GER conditions. The second framework, Variational Encoder Multi-Decoder (VE-MD), learns a shared latent space for emotion classification and structural representation prediction, including body and face cues. Two decoding strategies, DETR-based and heatmap-based, are explored to analyze the role of structural representations in group and individual settings. The thesis makes three main contributions: it clarifies the role of multimodality and structural cues in group-level affective computing; introduces two architectures for privacy-preserving multimodal GER; and shows that competitive performance can be achieved without using individual features as input data.