# CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization

arXiv: https://arxiv.org/abs/2603.16966 · HTML: https://mlsanity.com/paper/2603.16966/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks](https://mlsanity.com/paper/2012.14952/) (2020)
- [Fully Supervised Speaker Diarization](https://mlsanity.com/paper/1810.04719/) (2018)
- [The Second DIHARD Diarization Challenge: Dataset, task, and baselines](https://mlsanity.com/paper/1906.07839/) (2019)
- [Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap](https://mlsanity.com/paper/2003.02405/) (2020)
- [MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding](https://mlsanity.com/paper/2106.01541/) (2021)
- [From Speech-to-Speech Translation to Automatic Dubbing](https://mlsanity.com/paper/2001.06785/) (2020)
- [AVA-AVD: Audio-Visual Speaker Diarization in the Wild](https://mlsanity.com/paper/2111.14448/) (2021)
- [Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion](https://mlsanity.com/paper/1603.09725/) (2016)
- [Who said that?: Audio-visual speaker diarisation of real-world meetings](https://mlsanity.com/paper/1906.10042/) (2019)
- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency](https://mlsanity.com/paper/2406.02167/) (2024)
- [BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications](https://mlsanity.com/paper/2110.05781/) (2021)
- [Multimodal Speaker Segmentation and Diarization using Lexical and Acoustic Cues via Sequence to Sequence Neural Networks](https://mlsanity.com/paper/1805.10731/) (2018)
- [A Real-time Speaker Diarization System Based on Spatial Spectrum](https://mlsanity.com/paper/2107.09321/) (2021)
- [Multimodal Clustering with Role Induced Constraints for Speaker Diarization](https://mlsanity.com/paper/2204.00657/) (2022)
- [Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization](https://mlsanity.com/paper/2305.12927/) (2023)
- [Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning](https://mlsanity.com/paper/2403.15469/) (2024)
- [Improving Speaker Diarization using Semantic Information: Joint Pairwise Constraints Propagation](https://mlsanity.com/paper/2309.10456/) (2023)
- [Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization](https://mlsanity.com/paper/2508.08550/) (2025)
- [Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling](https://mlsanity.com/paper/2602.00597/) (2026)

## Abstract (arXiv)

Traditional speaker diarization systems have primarily focused on constrained scenarios such as meetings and interviews, where the number of speakers is limited and acoustic conditions are relatively clean. To explore open-world speaker diarization, we extend this task to the visual media domain, encompassing complex audiovisual programs such as films and TV series. This new setting introduces several challenges, including long-form video understanding, a large number of speakers, cross-modal asynchrony between audio and visual cues, and uncontrolled in-the-wild variability. To address these challenges, we propose Cinematic Speaker Registration & Diarization (CineSRD), a unified multimodal framework that leverages visual, acoustic, and linguistic cues from video, speech, and subtitles for speaker annotation. CineSRD first performs visual anchor clustering to register initial speakers and then integrates an audio language model for speaker turn detection, refining annotations and supplementing unregistered off-screen speakers. Furthermore, we construct and release a dedicated speaker diarization benchmark for visual media that includes Chinese and English programs. Experimental results demonstrate that CineSRD achieves superior performance on the proposed benchmark and competitive results on conventional datasets, validating its robustness and generalizability in open-world visual media settings.