# EgoNormia: Benchmarking Physical Social Norm Understanding

arXiv: https://arxiv.org/abs/2502.20490 · HTML: https://mlsanity.com/paper/2502.20490/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents](https://mlsanity.com/paper/2403.08715/) (2024)
- [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024)
- [NormBank: A Knowledge Bank of Situational Social Norms](https://mlsanity.com/paper/2305.17008/) (2023)
- [Principles and Guidelines for Evaluating Social Robot Navigation Algorithms](https://mlsanity.com/paper/2306.16740/) (2023)
- [ContPhy: Continuum Physical Concept Learning and Reasoning from Videos](https://mlsanity.com/paper/2402.06119/) (2024)
- [Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events](https://mlsanity.com/paper/2412.05725/) (2024)

## Built on by (most cited first, 13 of 13)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learning](https://mlsanity.com/paper/2503.16965/) (2025)
- [Using large language models for embodied planning introduces systematic safety risks](https://mlsanity.com/paper/2604.18463/) (2026)
- [VideoNorms: Benchmarking Cultural Awareness of Video Language Models](https://mlsanity.com/paper/2510.08543/) (2025)
- [HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes](https://mlsanity.com/paper/2508.13692/) (2025)
- [Social Caption: Evaluating Social Understanding in Multimodal Models](https://mlsanity.com/paper/2601.14569/) (2026)
- [NormAct: Benchmarking Embodied Agents' Proactive Compliance with Unspoken Social Norms](https://mlsanity.com/paper/2606.27826/) (2026)
- [Social Norm Reasoning in Multimodal Language Models: An Evaluation](https://mlsanity.com/paper/2603.03590/) (2026)
- [VIVA+: Human-Centered Situational Decision-Making](https://mlsanity.com/paper/2509.23698/) (2025)
- [Act2See: Emergent Active Visual Perception for Video Reasoning](https://mlsanity.com/paper/2605.01657/) (2026)
- [Are LLMs Bad at Moral Reasoning?](https://mlsanity.com/paper/2606.11635/) (2026)
- [EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes](https://mlsanity.com/paper/2607.00547/) (2026)
- [RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy](https://mlsanity.com/paper/2609.21371/) (2026)

## Abstract (arXiv)

Human activity is moderated by norms; however, supervision for normative reasoning is sparse, particularly where norms are physically- or socially-grounded. We thus present EGONORMIA |ε|, comprising 1,853 (200 for EGONORMIA-verified) multiple choice questions (MCQs) grounded within egocentric videos of human interactions, enabling the evaluation and improvement of normative reasoning in vision-language models (VLMs). EGONORMIA spans seven norm categories: safety, privacy, proxemics, politeness, cooperation, coordination/proactivity, and communication/legibility. To compile this dataset at scale, we propose a novel pipeline to generate grounded MCQs from raw egocentric video. Our work demonstrates that current state-of-the-art VLMs lack robust grounded norm understanding, scoring a maximum of 54% on EGONORMIA and 65% on EGONORMIA-verified, with performance across norm categories indicating significant risks of safety and privacy when VLMs are used in real-world agents. We additionally explore methods for improving normative understanding, demonstrating that a naive retrieval-based generation (RAG) method using EGONORMIA can enhance normative reasoning in VLMs.