# Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge

arXiv: https://arxiv.org/abs/2506.02621 · HTML: https://mlsanity.com/paper/2506.02621/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models.
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- Speech and audio language models / [Speaker recognition and diarization](https://mlsanity.com/speech/speaker-recognition-and-diarization/)

## Builds on (most cited first, 11 of 11)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-End Neural Speaker Diarization with Self-attention](https://mlsanity.com/paper/1909.06247/) (2019)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [CN-CELEB: a challenging Chinese speaker recognition dataset](https://mlsanity.com/paper/1911.01799/) (2019)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [Lip-reading with Densely Connected Temporal Convolutional Networks](https://mlsanity.com/paper/2009.14233/) (2020)
- [Spatial-Temporal Transformer for 3D Point Cloud Sequences](https://mlsanity.com/paper/2110.09783/) (2021)
- [Third DIHARD Challenge Evaluation Plan](https://mlsanity.com/paper/2006.05815/) (2020)
- [Augment on Manifold: Mixup Regularization with UMAP](https://mlsanity.com/paper/2312.13141/) (2023)

## Abstract (arXiv)

This paper presents the system developed for Task 1 of the Multi-modal Information-based Speech Processing (MISP) 2025 Challenge. We introduce CASA-Net, an embedding fusion method designed for end-to-end audio-visual speaker diarization (AVSD) systems. CASA-Net incorporates a cross-attention (CA) module to effectively capture cross-modal interactions in audio-visual signals and employs a self-attention (SA) module to learn contextual relationships among audio-visual frames. To further enhance performance, we adopt a training strategy that integrates pseudo-label refinement and retraining, improving the accuracy of timestamp predictions. Additionally, median filtering and overlap averaging are applied as post-processing techniques to eliminate outliers and smooth prediction labels. Our system achieved a diarization error rate (DER) of 8.18% on the evaluation set, representing a relative improvement of 47.3% over the baseline DER of 15.52%.