# A Domain Based Approach to Social Relation Recognition

arXiv: https://arxiv.org/abs/1704.06456 · HTML: https://mlsanity.com/paper/1704.06456/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2017.

## Built on by (most cited first, 18 of 18)

- [Perceptual Indistinguishability-Net (PI-Net): Facial Image Obfuscation with Manipulable Semantics](https://mlsanity.com/paper/2104.01753/) (2021)
- [SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning](https://mlsanity.com/paper/2506.05425/) (2025)
- [CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks](https://mlsanity.com/paper/2103.04032/) (2021)
- [Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants](https://mlsanity.com/paper/2501.01243/) (2025)
- [Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark](https://mlsanity.com/paper/2109.11243/) (2021)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Human-to-Human Interaction Detection](https://mlsanity.com/paper/2307.00464/) (2023)
- [MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding](https://mlsanity.com/paper/2409.06224/) (2024)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Subversive Characters and Stereotyping Readers: Characterizing Queer Relationalities with Dialogue-Based Relation Extraction](https://mlsanity.com/paper/2410.14978/) (2024)
- [Interaction-GCN: A Graph Convolutional Network based framework for social interaction recognition in egocentric videos](https://mlsanity.com/paper/2104.14007/) (2021)
- [Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classification](https://mlsanity.com/paper/2402.19339/) (2024)
- [Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation](https://mlsanity.com/paper/2107.14425/) (2021)
- [Humans Social Relationship Classification during Accompaniment](https://mlsanity.com/paper/2207.02890/) (2022)
- [Face Clustering for Connection Discovery from Event Images](https://mlsanity.com/paper/2309.01092/) (2023)
- [From a Social Cognitive Perspective: Context-aware Visual Social Relationship Recognition](https://mlsanity.com/paper/2406.08358/) (2024)
- [Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification](https://mlsanity.com/paper/2605.28604/) (2026)
- [FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models](https://mlsanity.com/paper/2607.29602/) (2026)

## Abstract (arXiv)

Social relations are the foundation of human daily life. Developing techniques to analyze such relations from visual data bears great potential to build machines that better understand us and are capable of interacting with us at a social level. Previous investigations have remained partial due to the overwhelming diversity and complexity of the topic and consequently have only focused on a handful of social relations. In this paper, we argue that the domain-based theory from social psychology is a great starting point to systematically approach this problem. The theory provides coverage of all aspects of social relations and equally is concrete and predictive about the visual attributes and behaviors defining the relations included in each domain. We provide the first dataset built on this holistic conceptualization of social life that is composed of a hierarchical label space of social domains and social relations. We also contribute the first models to recognize such domains and relations and find superior performance for attribute based features. Beyond the encouraging performance of the attribute based approach, we also find interpretable features that are in accordance with the predictions from social psychology literature. Beyond our findings, we believe that our contributions more tightly interleave visual recognition and social psychology theory that has the potential to complement the theoretical work in the area with empirical and data-driven models of social life.