# Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion

arXiv: https://arxiv.org/abs/1811.10813 · HTML: https://mlsanity.com/paper/1811.10813/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2018.

## Built on by (most cited first, 21 of 21)

- [AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models](https://mlsanity.com/paper/2410.18325/) (2024)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Audio-Visual Person-of-Interest DeepFake Detection](https://mlsanity.com/paper/2204.03083/) (2022)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport](https://mlsanity.com/paper/2401.15489/) (2024)
- [CAD -- Contextual Multi-modal Alignment for Dynamic AVQA](https://mlsanity.com/paper/2310.16754/) (2023)
- [Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT](https://mlsanity.com/paper/2205.07180/) (2022)
- [CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition](https://mlsanity.com/paper/2305.16049/) (2023)
- [Audio-Visual Speaker Verification via Joint Cross-Attention](https://mlsanity.com/paper/2309.16569/) (2023)
- [Fusion of Embeddings Networks for Robust Combination of Text Dependent and Independent Speaker Recognition](https://mlsanity.com/paper/2106.10169/) (2021)
- [Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention](https://mlsanity.com/paper/2403.04654/) (2024)
- [Dynamic Cross Attention for Audio-Visual Person Verification](https://mlsanity.com/paper/2403.04661/) (2024)
- [Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition](https://mlsanity.com/paper/2408.09035/) (2024)
- [Learning Audio-Visual embedding for Person Verification in the Wild](https://mlsanity.com/paper/2209.04093/) (2022)
- [Discriminative Semantic Transitive Consistency for Cross-Modal Learning](https://mlsanity.com/paper/2103.14103/) (2021)
- [Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification](https://mlsanity.com/paper/2309.07115/) (2023)
- [A Study of Multimodal Person Verification Using Audio-Visual-Thermal Data](https://mlsanity.com/paper/2110.12136/) (2021)
- [On the Behavior of Audio-Visual Fusion Architectures in Identity Verification Tasks](https://mlsanity.com/paper/2311.05071/) (2023)
- [SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification](https://mlsanity.com/paper/2506.17694/) (2025)

## Abstract (arXiv)

In this paper, we present a multi-modal online person verification system using both speech and visual signals. Inspired by neuroscientific findings on the association of voice and face, we propose an attention-based end-to-end neural network that learns multi-sensory associations for the task of person verification. The attention mechanism in our proposed network learns to conditionally select a salient modality between speech and facial representations that provides a balance between complementary inputs. By virtue of this capability, the network is robust to missing or corrupted data from either modality. In the VoxCeleb2 dataset, we show that our method performs favorably against competing multi-modal methods. Even for extreme cases of large corruption or an entirely missing modality, our method demonstrates robustness over other unimodal methods.