# Learning Interactions and Relationships between Movie Characters

arXiv: https://arxiv.org/abs/2003.13158 · HTML: https://mlsanity.com/paper/2003.13158/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2020.

## Built on by (most cited first, 22 of 22)

- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [Face, Body, Voice: Video Person-Clustering with Multiple Modalities](https://mlsanity.com/paper/2105.09939/) (2021)
- [Social Fabric: Tubelet Compositions for Video Relation Detection](https://mlsanity.com/paper/2108.08363/) (2021)
- [Contextual AD Narration with Interleaved Multimodal Sequence](https://mlsanity.com/paper/2403.12922/) (2024)
- [Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos](https://mlsanity.com/paper/2208.01954/) (2022)
- [SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues](https://mlsanity.com/paper/2106.01006/) (2021)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark](https://mlsanity.com/paper/2109.11243/) (2021)
- [Visual Objectification in Films: Towards a New AI Task for Video Interpretation](https://mlsanity.com/paper/2401.13296/) (2024)
- [Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation](https://mlsanity.com/paper/2306.10122/) (2023)
- [FocusedAD: Character-centric Movie Audio Description](https://mlsanity.com/paper/2504.12157/) (2025)
- [Listen Then See: Video Alignment with Speaker Attention](https://mlsanity.com/paper/2404.13530/) (2024)
- [Self-Contained Entity Discovery from Captioned Videos](https://mlsanity.com/paper/2208.06662/) (2022)
- [Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM](https://mlsanity.com/paper/2409.09362/) (2024)
- [LAEO-Net++: revisiting people Looking At Each Other in videos](https://mlsanity.com/paper/2101.02136/) (2021)
- [Movie Genre Classification by Language Augmentation and Shot Sampling](https://mlsanity.com/paper/2203.13281/) (2022)
- [From a Social Cognitive Perspective: Context-aware Visual Social Relationship Recognition](https://mlsanity.com/paper/2406.08358/) (2024)
- [CloSE: A Geometric Shape-Agnostic Cloth State Representation](https://mlsanity.com/paper/2504.05033/) (2025)
- [StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Dataset](https://mlsanity.com/paper/2606.06338/) (2026)

## Abstract (arXiv)

Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.