# Gems: Group Emotion Profiling Through Multimodal Situational Understanding

arXiv: https://arxiv.org/abs/2507.22393 · HTML: https://mlsanity.com/paper/2507.22393/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 13 of 13)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Prediction and Localization of Student Engagement in the Wild](https://mlsanity.com/paper/1804.00858/) (2018)
- [EmoLLM: Multimodal Emotional Understanding Meets Large Language Models](https://mlsanity.com/paper/2406.16442/) (2024)
- [EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction](https://mlsanity.com/paper/1808.07773/) (2018)
- [Predicting Group Cohesiveness in Images](https://mlsanity.com/paper/1812.11771/) (2018)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [A Survey of Deep Learning for Group-level Emotion Recognition](https://mlsanity.com/paper/2408.15276/) (2024)
- [Emolysis: A Multimodal Open-Source Group Emotion Analysis and Visualization Toolkit](https://mlsanity.com/paper/2305.05255/) (2023)

## Built on by (most cited first, 1 of 1)

- [CSGaze: Context-aware Social Gaze Prediction](https://mlsanity.com/paper/2511.05955/) (2025)

## Abstract (arXiv)

Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset. GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS