# ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild

arXiv: https://arxiv.org/abs/2205.05177 · HTML: https://mlsanity.com/paper/2205.05177/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 92 references that are in the map.
- First posted on arXiv 10 May 2022.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification](https://mlsanity.com/paper/2102.00457/) (2021)
- [Learning Delicate Local Representations for Multi-Person Pose Estimation](https://mlsanity.com/paper/2003.04030/) (2020)
- [Rethinking on Multi-Stage Networks for Human Pose Estimation](https://mlsanity.com/paper/1901.00148/) (2019)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour](https://mlsanity.com/paper/1801.06055/) (2018)
- [Quo Vadis, Skeleton Action Recognition ?](https://mlsanity.com/paper/2007.02072/) (2020)
- [F-formation Detection: Individuating Free-standing Conversational Groups in Images](https://mlsanity.com/paper/1409.2702/) (2014)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [Defining and Quantifying Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2009.12842/) (2020)
- [Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cues](https://mlsanity.com/paper/2107.13576/) (2021)
- [A Modular Approach for Synchronized Wireless Multimodal Multisensor Data Acquisition in Highly Dynamic Social Settings](https://mlsanity.com/paper/2008.03715/) (2020)
- [Towards automatic estimation of conversation floors within F-formations](https://mlsanity.com/paper/1907.10384/) (2019)
- [Detecting F-formations & Roles in Crowded Social Scenes with Wearables: Combining Proxemics & Dynamics using LSTMs](https://mlsanity.com/paper/1911.07279/) (2019)

## Built on by (most cited first, 10 of 10)

- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos](https://mlsanity.com/paper/2407.09503/) (2024)
- [PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction](https://mlsanity.com/paper/2604.08125/) (2026)
- [EgoSocial: Benchmarking Proactive Intervention Ability of Omnimodal LLMs via Egocentric Social Interaction Perception](https://mlsanity.com/paper/2510.13105/) (2025)
- [A Synchronized Audio-Visual Multi-View Capture System](https://mlsanity.com/paper/2603.23089/) (2026)
- [COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention](https://mlsanity.com/paper/2607.28649/) (2026)
- [Where Should I Join? Robot Group Joining via Language-Guided Goal Prediction](https://mlsanity.com/paper/2609.28467/) (2026)
- [Generative Interactions: Weaving Multiparty Human Motion with Bilevel Latent Dynamics](https://mlsanity.com/paper/2609.37708/) (2026)
- [Breakdown of Conversational Involvement: A Joint Multi-Modal Analysis of Conversation Floors and F-formation Detection with Multiple Features](https://mlsanity.com/paper/2610.18627/) (2026)

## Abstract (arXiv)

Recording the dynamics of unscripted human interactions in the wild is challenging due to the delicate trade-offs between several factors: participant privacy, ecological validity, data fidelity, and logistical overheads. To address these, following a 'datasets for the community by the community' ethos, we propose the Conference Living Lab (ConfLab): a new concept for multimodal multisensor data collection of in-the-wild free-standing social conversations. For the first instantiation of ConfLab described here, we organized a real-life professional networking event at a major international conference. Involving 48 conference attendees, the dataset captures a diverse mix of status, acquaintance, and networking motivations. Our capture setup improves upon the data fidelity of prior in-the-wild datasets while retaining privacy sensitivity: 8 videos (1920x1080, 60 fps) from a non-invasive overhead view, and custom wearable sensors with onboard recording of body motion (full 9-axis IMU), privacy-preserving low-frequency audio (1250 Hz), and Bluetooth-based proximity. Additionally, we developed custom solutions for distributed hardware synchronization at acquisition and time-efficient continuous annotation of body keypoints and actions at high sampling rates. Our benchmarks showcase some of the open research tasks related to in-the-wild privacy-preserving social data analysis: keypoints detection from overhead camera views, skeleton-based no-audio speaker detection, and F-formation detection.