mlsanityData to 19 Oct 2026

Audio-Visual Speaker Diarization: Current Databases, Approaches and Challenges

arXiv 2409.05659 ↗First posted 9 Sep 2024cs.SD cs.MM eess.AScited by 1

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

Nowadays, the large amount of audio-visual content available has fostered the need to develop new robust automatic speaker diarization systems to analyse and characterise it. This kind of system helps to reduce the cost of doing this process manually and allows the use of the speaker information for different applications, as a huge quantity of information is present, for example, images of faces, or audio recordings. Therefore, this paper aims to address a critical area in the field of speaker diarization systems, the integration of audio-visual content of different domains. This paper seeks to push beyond current state-of-the-art practices by developing a robust audio-visual speaker diarization framework adaptable to various data domains, including TV scenarios, meetings, and daily activities. Unlike most of the existing audio-visual speaker diarization systems, this framework will also include the proposal of an approach to lead the precise assignment of specific identities in TV scenarios where celebrities appear. In addition, in this work, we have conducted an extensive compilation of the current state-of-the-art approaches and the existing databases for developing audio-visual speaker diarization.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 51 of its 123 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

51 of 123 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization16 Jan
  2. 2023VoxBlink: A Large Scale Speaker Verification Dataset on Camera14 Aug
  3. 2022Using Active Speaker Faces for Diarization in TV shows30 Mar
  4. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wild29 Nov
  5. 2021ECAPA-TDNN Embeddings for Speaker Diarization3 Apr
  6. 2021A Review of Speaker Diarization: Recent Advances with Deep Learning24 Jan

Other references · 45

  1. 2018YOLOv3: An Incremental Improvementcited by 2,687
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,574
  3. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 454
  4. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 221
  5. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 190
  6. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 158
  7. 2020Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard taskscited by 130
  8. 2020Spot the conversation: speaker diarisation in the wildcited by 128
  9. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 125
  10. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 89
  11. 2020Active Speakers in Contextcited by 42
  12. 2021EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environmentscited by 41
33 more
  1. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 37
  2. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 32
  3. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 31
  4. 2016Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusioncited by 29
  5. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 29
  6. 2019Who said that?: Audio-visual speaker diarisation of real-world meetingscited by 28
  7. 2022End-to-End Active Speaker Detectioncited by 26
  8. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 26
  9. 2021Face, Body, Voice: Video Person-Clustering with Multiple Modalitiescited by 25
  10. 2016Cross-modal Supervision for Learning Active Speaker Detection in Videocited by 22
  11. 2021Look Who's Talking: Active Speaker Detection in the Wildcited by 22
  12. 2018From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Scriptcited by 21
  13. 2023The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognitioncited by 18
  14. 2020Self-supervised learning for audio-visual speaker diarizationcited by 17
  15. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 17
  16. 2020Online Multi-modal Person Search in Videoscited by 11
  17. 2021Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Trackingcited by 10
  18. 2022Rethinking Audio-visual Synchronization for Active Speaker Detectioncited by 10
  19. 2022Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarizationcited by 10
  20. 2024Look, Listen and Recognise: Character-Aware Audio-Visual Subtitlingcited by 9
  21. 2020Cross modal video representations for weakly supervised active speaker localizationcited by 7
  22. 2023WASD: A Wilder Active Speaker Detection Datasetcited by 7
  23. 2023STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarizationcited by 7
  24. 2018Audiovisual speaker diarization of TV seriescited by 6
  25. 2020Serial Speakers: a Dataset of TV Seriescited by 5
  26. 2021FaVoA: Face-Voice Association Favours Ambiguous Speaker Detectioncited by 5
  27. 2022Audio visual character profiles for detecting background characters in entertainment mediacited by 5
  28. 2022Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detectioncited by 5
  29. 2019Audiovisual Speaker Tracking using Nonlinear Dynamical Systems with Dynamic Stream Weightscited by 4
  30. 2023Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directionscited by 4
  31. 2023Audio Visual Speaker Localization from EgoCentric Viewscited by 3
  32. 2014Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regressioncited by 2
  33. 2022Unsupervised active speaker detection in media content using cross-modal informationcited by 2

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization
  2. 2023Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
  3. 2022Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
  4. 2026CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
  5. 2023USED: Universal Speaker Extraction and Diarization
  6. 2024A Review of Common Online Speaker Diarization Methods
  7. 20243D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
  8. 2024Character-aware audio-visual subtitling in context