# Deep Multimodal Speaker Naming

arXiv: https://arxiv.org/abs/1507.04831 · HTML: https://mlsanity.com/paper/1507.04831/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2015.

## Built on by (most cited first, 5 of 5)

- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [Character-aware audio-visual subtitling in context](https://mlsanity.com/paper/2410.11068/) (2024)
- [No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration](https://mlsanity.com/paper/2211.00549/) (2022)
- [REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild](https://mlsanity.com/paper/2403.01229/) (2024)
- [Multilingual Audio-Visual Smartphone Dataset And Evaluation](https://mlsanity.com/paper/2109.04138/) (2021)

## Abstract (arXiv)

Automatic speaker naming is the problem of localizing as well as identifying each speaking character in a TV/movie/live show video. This is a challenging problem mainly attributes to its multimodal nature, namely face cue alone is insufficient to achieve good performance. Previous multimodal approaches to this problem usually process the data of different modalities individually and merge them using handcrafted heuristics. Such approaches work well for simple scenes, but fail to achieve high performance for speakers with large appearance variations. In this paper, we propose a novel convolutional neural networks (CNN) based learning framework to automatically learn the fusion function of both face and audio cues. We show that without using face tracking, facial landmark localization or subtitle/transcript, our system with robust multimodal feature extraction is able to achieve state-of-the-art speaker naming performance evaluated on two diverse TV series. The dataset and implementation of our algorithm are publicly available online.