mlsanityData to 19 Oct 2026

Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge

arXiv 2506.02621 ↗First posted 3 Jun 2025cs.SDcited by 0

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

This paper presents the system developed for Task 1 of the Multi-modal Information-based Speech Processing (MISP) 2025 Challenge. We introduce CASA-Net, an embedding fusion method designed for end-to-end audio-visual speaker diarization (AVSD) systems. CASA-Net incorporates a cross-attention (CA) module to effectively capture cross-modal interactions in audio-visual signals and employs a self-attention (SA) module to learn contextual relationships among audio-visual frames. To further enhance performance, we adopt a training strategy that integrates pseudo-label refinement and retraining, improving the accuracy of timestamp predictions. Additionally, median filtering and overlap averaging are applied as post-processing techniques to eliminate outliers and smooth prediction labels. Our system achieved a diarization error rate (DER) of 8.18% on the evaluation set, representing a relative improvement of 47.3% over the baseline DER of 15.52%.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 11 of its 19 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

11 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021A Review of Speaker Diarization: Recent Advances with Deep Learning24 Jan

Other references · 10

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,243
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  3. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 139
  4. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 133
  5. 2019CN-CELEB: a challenging Chinese speaker recognition datasetcited by 123
  6. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 60
  7. 2020Lip-reading with Densely Connected Temporal Convolutional Networkscited by 28
  8. 2021Spatial-Temporal Transformer for 3D Point Cloud Sequencescited by 23
  9. 2020Third DIHARD Challenge Evaluation Plancited by 22
  10. 2023Augment on Manifold: Mixup Regularization with UMAPcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization
  2. 2023Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
  3. 2023Improving Transformer-based End-to-End Speaker Diarization by Assigning Auxiliary Losses to Attention Heads
  4. 2023Semi-supervised multi-channel speaker diarization with cross-channel attention
  5. 2022Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
  6. 2024Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios
  7. 2023Attention-based Encoder-Decoder End-to-End Neural Diarization with Embedding Enhancer
  8. 2024From Modular to End-to-End Speaker Diarization