# Character-Centric Understanding of Animated Movies

arXiv: https://arxiv.org/abs/2509.12204 · HTML: https://mlsanity.com/paper/2509.12204/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 70 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation](https://mlsanity.com/paper/2406.10082/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description](https://mlsanity.com/paper/2407.15850/) (2024)
- [Who said that?: Audio-visual speaker diarisation of real-world meetings](https://mlsanity.com/paper/1906.10042/) (2019)
- [Learning Video Context as Interleaved Multimodal Sequences](https://mlsanity.com/paper/2407.21757/) (2024)
- [LLM-AD: Large Language Model based Audio Description System](https://mlsanity.com/paper/2405.00983/) (2024)
- [Object Detection for Comics using Manga109 Annotations](https://mlsanity.com/paper/1803.08670/) (2018)
- [Contextual AD Narration with Interleaved Multimodal Sequence](https://mlsanity.com/paper/2403.12922/) (2024)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)
- [Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names](https://mlsanity.com/paper/2408.00298/) (2024)
- [DAF:re: A Challenging, Crowd-Sourced, Large-Scale, Long-Tailed Dataset For Anime Character Recognition](https://mlsanity.com/paper/2101.08674/) (2021)
- [Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://mlsanity.com/paper/2308.09096/) (2023)
- [Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling](https://mlsanity.com/paper/2401.12039/) (2024)
- [ACFD: Asymmetric Cartoon Face Detector](https://mlsanity.com/paper/2007.00899/) (2020)

## Built on by (most cited first, 3 of 3)

- [VideoNSA: Native Sparse Attention Scales Video Understanding](https://mlsanity.com/paper/2510.02295/) (2025)
- [Measuring Gender Representation in Animated Films](https://mlsanity.com/paper/2608.21429/) (2026)
- [StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos](https://mlsanity.com/paper/2608.12549/) (2026)

## Abstract (arXiv)

Animated movies are captivating for their unique character designs and imaginative storytelling, yet they pose significant challenges for existing recognition systems. Unlike the consistent visual patterns detected by conventional face recognition methods, animated characters exhibit extreme diversity in their appearance, motion, and deformation. In this work, we propose an audio-visual pipeline to enable automatic and robust animated character recognition, and thereby enhance character-centric understanding of animated movies. Central to our approach is the automatic construction of an audio-visual character bank from online sources. This bank contains both visual exemplars and voice (audio) samples for each character, enabling subsequent multi-modal character recognition despite long-tailed appearance distributions. Building on accurate character recognition, we explore two downstream applications: Audio Description (AD) generation for visually impaired audiences, and character-aware subtitling for the hearing impaired. To support research in this domain, we introduce CMD-AM, a new dataset of 75 animated movies with comprehensive annotations. Our character-centric pipeline demonstrates significant improvements in both accessibility and narrative comprehension for animated content over prior face-detection-based approaches. For the code and dataset, visit https://www.robots.ox.ac.uk/~vgg/research/animated_ad/.