# SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

arXiv: https://arxiv.org/abs/1506.06882 · HTML: https://mlsanity.com/paper/1506.06882/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2015.

## Built on by (most cited first, 27 of 27)

- [MMPTRACK: Large-scale Densely Annotated Multi-camera Multiple People Tracking Benchmark](https://mlsanity.com/paper/2111.15157/) (2021)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Nonverbal Interaction Detection](https://mlsanity.com/paper/2407.08133/) (2024)
- [JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups](https://mlsanity.com/paper/2404.04458/) (2024)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild](https://mlsanity.com/paper/2205.05177/) (2022)
- [SocialInteractionGAN: Multi-person Interaction Sequence Generation](https://mlsanity.com/paper/2103.05916/) (2021)
- [Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild](https://mlsanity.com/paper/2211.00794/) (2022)
- [A Flexible Multi-view Multi-modal Imaging System for Outdoor Scenes](https://mlsanity.com/paper/2302.10465/) (2023)
- [Detecting socially interacting groups using f-formation: A survey of taxonomy, methods, datasets, applications, challenges, and future research directions](https://mlsanity.com/paper/2108.06181/) (2021)
- [GROWL: Group Detection With Link Prediction](https://mlsanity.com/paper/2111.04397/) (2021)
- [Conversation Group Detection With Spatio-Temporal Context](https://mlsanity.com/paper/2206.02559/) (2022)
- [Inferred vs traditional personality assessment: are we predicting the same thing?](https://mlsanity.com/paper/2103.09632/) (2021)
- [No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration](https://mlsanity.com/paper/2211.00549/) (2022)
- [REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild](https://mlsanity.com/paper/2403.01229/) (2024)
- [LAEO-Net++: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/2101.02136/) (2021)
- [Retail store customer behavior analysis system: Design and Implementation](https://mlsanity.com/paper/2309.03232/) (2023)
- [PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits](https://mlsanity.com/paper/2509.11362/) (2025)
- [3D Ground Truth Reconstruction from Multi-Camera Annotations Using UKF](https://mlsanity.com/paper/2511.17609/) (2025)
- [EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild](https://mlsanity.com/paper/2603.22249/) (2026)
- [GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos](https://mlsanity.com/paper/2604.16214/) (2026)
- [Gaze-Anchored Social Net: Decoding Implicit Relations via Joint Modeling](https://mlsanity.com/paper/2607.22847/) (2026)
- [COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention](https://mlsanity.com/paper/2607.28649/) (2026)
- [Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations](https://mlsanity.com/paper/2608.23344/) (2026)
- [Pseudo-Label Augmentation for Affect Sensing in Small Collaborative Groups](https://mlsanity.com/paper/2609.16077/) (2026)
- [Where Should I Join? Robot Group Joining via Language-Guided Goal Prediction](https://mlsanity.com/paper/2609.28467/) (2026)

## Abstract (arXiv)

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.