# Audio-Visual Speaker Verification via Joint Cross-Attention

arXiv: https://arxiv.org/abs/2309.16569 · HTML: https://mlsanity.com/paper/2309.16569/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention](https://mlsanity.com/paper/2008.06581/) (2020)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020)
- [Visual Question Answering with Memory-Augmented Networks](https://mlsanity.com/paper/1707.04968/) (2017)
- [Audio-visual Speaker Recognition with a Cross-modal Discriminative Network](https://mlsanity.com/paper/2008.03894/) (2020)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Disjoint Mapping Network for Cross-modal Matching of Voices and Faces](https://mlsanity.com/paper/1807.04836/) (2018)
- [Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion](https://mlsanity.com/paper/1811.10813/) (2018)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2111.05222/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Deep Multimodal Representation Learning from Temporal Data](https://mlsanity.com/paper/1704.03152/) (2017)
- [Xi-Vector Embedding for Speaker Recognition](https://mlsanity.com/paper/2108.05679/) (2021)
- [PCF: ECAPA-TDNN with Progressive Channel Fusion for Speaker Verification](https://mlsanity.com/paper/2303.00204/) (2023)
- [Multimodal Speech Emotion Recognition using Cross Attention with Aligned Audio and Text](https://mlsanity.com/paper/2207.12895/) (2022)
- [Speaker recognition with two-step multi-modal deep cleansing](https://mlsanity.com/paper/2210.15903/) (2022)
- [Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition](https://mlsanity.com/paper/2304.07958/) (2023)
- [Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification](https://mlsanity.com/paper/2302.11254/) (2023)
- [Learning Audio-Visual embedding for Person Verification in the Wild](https://mlsanity.com/paper/2209.04093/) (2022)

## Built on by (most cited first, 4 of 4)

- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention](https://mlsanity.com/paper/2403.04654/) (2024)
- [Dynamic Cross Attention for Audio-Visual Person Verification](https://mlsanity.com/paper/2403.04661/) (2024)
- [SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification](https://mlsanity.com/paper/2506.17694/) (2025)

## Abstract (arXiv)

Speaker verification has been widely explored using speech signals, which has shown significant improvement using deep models. Recently, there has been a surge in exploring faces and voices as they can offer more complementary and comprehensive information than relying only on a single modality of speech signals. Though current methods in the literature on the fusion of faces and voices have shown improvement over that of individual face or voice modalities, the potential of audio-visual fusion is not fully explored for speaker verification. Most of the existing methods based on audio-visual fusion either rely on score-level fusion or simple feature concatenation. In this work, we have explored cross-modal joint attention to fully leverage the inter-modal complementary information and the intra-modal information for speaker verification. Specifically, we estimate the cross-attention weights based on the correlation between the joint feature presentation and that of the individual feature representations in order to effectively capture both intra-modal as well inter-modal relationships among the faces and voices. We have shown that efficiently leveraging the intra- and inter-modal relationships significantly improves the performance of audio-visual fusion for speaker verification. The performance of the proposed approach has been evaluated on the Voxceleb1 dataset. Results show that the proposed approach can significantly outperform the state-of-the-art methods of audio-visual fusion for speaker verification.