# Face and Voice Cross-modal Association with Learning Convex Feature Embedding

arXiv: https://arxiv.org/abs/2607.28129 · HTML: https://mlsanity.com/paper/2607.28129/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 95 references that are in the map.
- First posted on arXiv 30 Jul 2026.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [PyTorch Metric Learning](https://mlsanity.com/paper/2008.09164/) (2020)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2107.14724/) (2021)
- [Universal Weighting Metric Learning for Cross-Modal Matching](https://mlsanity.com/paper/2010.03403/) (2020)

## Abstract (arXiv)

Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal attention mechanisms with convex embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.