# Exploring Robust Face-Voice Matching in Multilingual Environments

arXiv: https://arxiv.org/abs/2407.19875 · HTML: https://mlsanity.com/paper/2407.19875/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020)
- [Disjoint Mapping Network for Cross-modal Matching of Voices and Faces](https://mlsanity.com/paper/1807.04836/) (2018)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals](https://mlsanity.com/paper/1909.08685/) (2019)
- [Speech-based Age and Gender Prediction with Transformers](https://mlsanity.com/paper/2306.16962/) (2023)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast](https://mlsanity.com/paper/2204.14057/) (2022)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan](https://mlsanity.com/paper/2404.09342/) (2024)

## Built on by (most cited first, 1 of 1)

- [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026)

## Abstract (arXiv)

This paper presents Team Xaiofei's innovative approach to exploring Face-Voice Association in Multilingual Environments (FAME) at ACM Multimedia 2024. We focus on the impact of different languages in face-voice matching by building upon Fusion and Orthogonal Projection (FOP), introducing four key components: a dual-branch structure, dynamic sample pair weighting, robust data augmentation, and score polarization strategy. Our dual-branch structure serves as an auxiliary mechanism to better integrate and provide more comprehensive information. We also introduce a dynamic weighting mechanism for various sample pairs to optimize learning. Data augmentation techniques are employed to enhance the model's generalization across diverse conditions. Additionally, score polarization strategy based on age and gender matching confidence clarifies and accentuates the final results. Our methods demonstrate significant effectiveness, achieving an equal error rate (EER) of 20.07 on the V2-EH dataset and 21.76 on the V1-EU dataset.