# Multi-Granularity Reasoning for Social Relation Recognition from Images

arXiv: https://arxiv.org/abs/1901.03067 · HTML: https://mlsanity.com/paper/1901.03067/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2019.

## Built on by (most cited first, 8 of 8)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)
- [Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering](https://mlsanity.com/paper/2112.07270/) (2021)
- [SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models](https://mlsanity.com/paper/2512.01148/) (2025)
- [Enhancing Social Relation Inference with Concise Interaction Graph and Discriminative Scene Representation](https://mlsanity.com/paper/2107.14425/) (2021)
- [GRITv2: Efficient and Light-weight Social Relation Recognition](https://mlsanity.com/paper/2403.06895/) (2024)
- [From a Social Cognitive Perspective: Context-aware Visual Social Relationship Recognition](https://mlsanity.com/paper/2406.08358/) (2024)
- [CloSE: A Geometric Shape-Agnostic Cloth State Representation](https://mlsanity.com/paper/2504.05033/) (2025)

## Abstract (arXiv)

Discovering social relations in images can make machines better interpret the behavior of human beings. However, automatically recognizing social relations in images is a challenging task due to the significant gap between the domains of visual content and social relation. Existing studies separately process various features such as faces expressions, body appearance, and contextual objects, thus they cannot comprehensively capture the multi-granularity semantics, such as scenes, regional cues of persons, and interactions among persons and objects. To bridge the domain gap, we propose a Multi-Granularity Reasoning framework for social relation recognition from images. The global knowledge and mid-level details are learned from the whole scene and the regions of persons and objects, respectively. Most importantly, we explore the fine-granularity pose keypoints of persons to discover the interactions among persons and objects. Specifically, the pose-guided Person-Object Graph and Person-Pose Graph are proposed to model the actions from persons to object and the interactions between paired persons, respectively. Based on the graphs, social relation reasoning is performed by graph convolutional networks. Finally, the global features and reasoned knowledge are integrated as a comprehensive representation for social relation recognition. Extensive experiments on two public datasets show the effectiveness of the proposed framework.