# COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

arXiv: https://arxiv.org/abs/2607.28649 · HTML: https://mlsanity.com/paper/2607.28649/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 101 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2501.16566/) (2025)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset](https://mlsanity.com/paper/2506.22554/) (2025)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations](https://mlsanity.com/paper/2403.10943/) (2024)
- [Social Genome: Grounded Social Reasoning Abilities of Multimodal Models](https://mlsanity.com/paper/2502.15109/) (2025)
- [F-formation Detection: Individuating Free-standing Conversational Groups in Images](https://mlsanity.com/paper/1409.2702/) (2014)
- [PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions](https://mlsanity.com/paper/2112.02604/) (2021)
- [ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild](https://mlsanity.com/paper/2205.05177/) (2022)
- [Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.16427/) (2025)
- [How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks](https://mlsanity.com/paper/2505.04628/) (2025)
- [Conversation Group Detection With Spatio-Temporal Context](https://mlsanity.com/paper/2206.02559/) (2022)
- [A Valid Self-Report is Never Late, Nor is it Early: On Considering the "Right" Temporal Distance for Assessing Emotional Experience](https://mlsanity.com/paper/2302.02821/) (2023)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [Magnitude Matters: a Superior Class of Similarity Metrics for Holistic Semantic Understanding](https://mlsanity.com/paper/2509.19323/) (2025)
- [REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild](https://mlsanity.com/paper/2403.01229/) (2024)

## Abstract (arXiv)

COSI-Lab presents a multimodal multi-sensor dataset of a real interdisciplinary scientific workshop with 32 academics at an international conference. It captures ecologically valid social interactions in a weakly scripted setting consisting of two 30-minute mingling sessions with real professional and social consequences. We argue that intelligent systems could better handle subjective perceptions by modeling their multiplicity not as label noise but as an explainable perspective-driven reasoning process. We focus on the Apparent Intent Inference (AII) problem as determined by ex-situ observers and conceptualize intentions to be independent of the future. We contribute a novel annotation process for AII that accounts for a perceiver's own interpretative tendencies, quantitative and qualitative analyses of intent narratives with respect to diversity, plausibility and grounding, benchmark tasks for AII and relevant contextual factors, speech quality audio for all participants as well as privacy-preserving multi-modal environmental and wearable sensor data, enabling lexical and nonverbal behavior analysis; and the coupling of self-reported participant goals (30 minute to 3 hour) with annotated AII (seconds).