# EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild

arXiv: https://arxiv.org/abs/2603.22249 · HTML: https://mlsanity.com/paper/2603.22249/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 29 of 29)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [Sekai: A Video Dataset towards World Exploration](https://mlsanity.com/paper/2506.15675/) (2025)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [PANDA: A Gigapixel-level Human-centric Video Dataset](https://mlsanity.com/paper/2003.04852/) (2020)
- [Probabilistic Future Prediction for Video Scene Understanding](https://mlsanity.com/paper/2003.06409/) (2020)
- [Learning Pedestrian Group Representations for Multi-modal Trajectory Prediction](https://mlsanity.com/paper/2207.09953/) (2022)
- [Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos](https://mlsanity.com/paper/2007.02632/) (2020)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [Detect Anything 3D in the Wild](https://mlsanity.com/paper/2504.07958/) (2025)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models](https://mlsanity.com/paper/2407.00263/) (2024)
- [VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models](https://mlsanity.com/paper/2503.07575/) (2025)
- [Towards More Practical Group Activity Detection: A New Benchmark and Model](https://mlsanity.com/paper/2312.02878/) (2023)
- [Seeing Culture: A Benchmark for Visual Reasoning and Grounding](https://mlsanity.com/paper/2509.16517/) (2025)
- [Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement](https://mlsanity.com/paper/2502.16902/) (2025)
- [Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels](https://mlsanity.com/paper/2410.07500/) (2024)
- [Dynamic Group Detection using VLM-augmented Temporal Groupness Graph](https://mlsanity.com/paper/2509.04758/) (2025)
- [MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenes](https://mlsanity.com/paper/2509.13484/) (2025)
- [BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models](https://mlsanity.com/paper/2510.11178/) (2025)

## Abstract (arXiv)

Social group detection, or the identification of humans involved in reciprocal interpersonal interactions (e.g., family members, friends, and customers and merchants), is a crucial component of social intelligence needed for agents transacting in the world. The few existing benchmarks for social group detection are limited by low scene diversity and reliance on third-person camera sources (e.g., surveillance footage). Consequently, these benchmarks generally lack real-world evaluation on how groups form and evolve in diverse cultural contexts and unconstrained settings. To address this gap, we introduce EgoGroups, a first-person view dataset that captures social dynamics in cities around the world. EgoGroups spans 65 countries covering low, medium, and high-crowd settings under four weather/time-of-day conditions. We include dense human annotations for person and social groups, along with rich geographic and scene metadata. Using this dataset, we performed an extensive evaluation of state-of-the-art VLM/LLMs and supervised models on their group detection capabilities. We found several interesting findings, including VLMs and LLMs can outperform supervised baselines in a zero-shot setting, while crowd density and cultural regions clearly influence model performance.