mlsanityData to 19 Oct 2026

Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization

arXiv 2312.04131 ↗First posted 7 Dec 2023eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained supervised and self-supervised speech models for audio-visual speaker diarization. Specifically, we adopt supervised~(ResNet and ECAPA-TDNN) and self-supervised pre-trained models~(WavLM and HuBERT) as the speaker and audio embedding extractors in an end-to-end audio-visual speaker diarization~(AVSD) system. Then we explore the effectiveness of different frameworks, including Transformer, Conformer, and cross-attention mechanism, in the audio-visual decoder. To mitigate the degradation of performance caused by separate training, we jointly train the audio encoder, speaker encoder, and audio-visual decoder in the AVSD system. Experiments on the MISP dataset demonstrate that the proposed method achieves superior performance and obtained third place in MISP Challenge 2022.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 16 of its 30 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

16 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Using Active Speaker Faces for Diarization in TV shows30 Mar
  2. 2022MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification29 Mar

Other references · 14

  1. 2017Attention Is All You Needcited by 61,180
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,456
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,923
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 1,166
  8. 2017VoxCeleb: a large-scale speaker identification datasetcited by 985
  9. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  10. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 231
  11. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 125
  12. 2019CN-CELEB: a challenging Chinese speaker recognition datasetcited by 123
2 more
  1. 2016Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusioncited by 29
  2. 2019Who said that?: Audio-visual speaker diarisation of real-world meetingscited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
  2. 2024Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization
  3. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wild
  4. 2023The FlySpeech Audio-Visual Speaker Diarization System for MISP Challenge 2022
  5. 2025Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge
  6. 2024Audio-Visual Speaker Diarization: Current Databases, Approaches and Challenges
  7. 2024Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization
  8. 2022Self-supervised Speaker Diarization