# Identity-Aware Multi-Sentence Video Description

arXiv: https://arxiv.org/abs/2008.09791 · HTML: https://mlsanity.com/paper/2008.09791/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [Sports Video Analysis on Large-Scale Data](https://mlsanity.com/paper/2208.04897/) (2022)
- [Localized Symbolic Knowledge Distillation for Visual Commonsense Models](https://mlsanity.com/paper/2312.04837/) (2023)
- [Controlled Caption Generation for Images Through Adversarial Attacks](https://mlsanity.com/paper/2107.03050/) (2021)
- [MICap: A Unified Model for Identity-aware Movie Descriptions](https://mlsanity.com/paper/2405.11483/) (2024)
- [What You See is What You Ask: Evaluating Audio Descriptions](https://mlsanity.com/paper/2510.00808/) (2025)
- [Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning](https://mlsanity.com/paper/2507.20163/) (2025)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach](https://mlsanity.com/paper/2507.04815/) (2025)
- [Addressing the ID-Matching Challenge in Long Video Captioning](https://mlsanity.com/paper/2510.06973/) (2025)
- ["Previously on ..." From Recaps to Story Summarization](https://mlsanity.com/paper/2405.11487/) (2024)
- [One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition](https://mlsanity.com/paper/2604.23173/) (2026)
- [Identity-Aware Human-Object Interaction Motion Captioning](https://mlsanity.com/paper/2608.20690/) (2026)

## Abstract (arXiv)

Standard video and movie description tasks abstract away from person identities, thus failing to link identities across sentences. We propose a multi-sentence Identity-Aware Video Description task, which overcomes this limitation and requires to re-identify persons locally within a set of consecutive clips. We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips, when the video descriptions are given. Our proposed approach to this task leverages a Transformer architecture allowing for coherent joint prediction of multiple IDs. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description. We first generate multi-sentence video descriptions, and then apply our Fill-in the Identity model to establish links between the predicted person entities. To be able to tackle both tasks, we augment the Large Scale Movie Description Challenge (LSMDC) benchmark with new annotations suited for our problem statement. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works, and allows us to generate descriptions with locally re-identified people.