# FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection

arXiv: https://arxiv.org/abs/2109.00577 · HTML: https://mlsanity.com/paper/2109.00577/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Speech2Face: Learning the Face Behind a Voice](https://mlsanity.com/paper/1905.09773/) (2019)
- [Active Speakers in Context](https://mlsanity.com/paper/2005.09812/) (2020)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech](https://mlsanity.com/paper/2004.05830/) (2020)
- [Multimodal Target Speech Separation with Voice and Face References](https://mlsanity.com/paper/2005.08335/) (2020)

## Built on by (most cited first, 4 of 4)

- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge](https://mlsanity.com/paper/2211.15377/) (2022)
- [A trained humanoid robot can perform human-like crossmodal social attention and conflict resolution](https://mlsanity.com/paper/2111.01906/) (2021)
- [FabuLight-ASD: Unveiling Speech Activity via Body Language](https://mlsanity.com/paper/2411.13674/) (2024)

## Abstract (arXiv)

The strong relation between face and voice can aid active speaker detection systems when faces are visible, even in difficult settings, when the face of a speaker is not clear or when there are several people in the same scene. By being capable of estimating the frontal facial representation of a person from his/her speech, it becomes easier to determine whether he/she is a potential candidate for being classified as an active speaker, even in challenging cases in which no mouth movement is detected from any person in that same scene. By incorporating a face-voice association neural network into an existing state-of-the-art active speaker detection model, we introduce FaVoA (Face-Voice Association Ambiguous Speaker Detector), a neural network model that can correctly classify particularly ambiguous scenarios. FaVoA not only finds positive associations, but helps to rule out non-matching face-voice associations, where a face does not match a voice. Its use of a gated-bimodal-unit architecture for the fusion of those models offers a way to quantitatively determine how much each modality contributes to the classification.