# Fusion and Orthogonal Projection for Improved Face-Voice Association arXiv: https://arxiv.org/abs/2112.10483 ยท HTML: https://mlsanity.com/paper/2112.10483/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 21 references that are in the map. - First posted on arXiv 20 Dec 2021. ## Builds on (most cited first, 13 of 13) - [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015) - [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017) - [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017) - [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018) - [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018) - [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019) - [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021) - [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018) - [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020) - [Disjoint Mapping Network for Cross-modal Matching of Voices and Faces](https://mlsanity.com/paper/1807.04836/) (2018) - [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021) - [Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals](https://mlsanity.com/paper/1909.08685/) (2019) - [Git Loss for Deep Face Recognition](https://mlsanity.com/paper/1807.08512/) (2018) ## Built on by (most cited first, 29 of 29) - [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023) - [A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios](https://mlsanity.com/paper/2409.17864/) (2024) - [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023) - [PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association](https://mlsanity.com/paper/2505.17002/) (2025) - [Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plan](https://mlsanity.com/paper/2508.04592/) (2025) - [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025) - [Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan](https://mlsanity.com/paper/2404.09342/) (2024) - [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024) - [Learning Branched Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2208.10238/) (2022) - [DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement Estimation in Conversation](https://mlsanity.com/paper/2308.01966/) (2023) - [Guiding Attention using Partial-Order Relationships for Image Captioning](https://mlsanity.com/paper/2204.07476/) (2022) - [Learning Audio-Visual embedding for Person Verification in the Wild](https://mlsanity.com/paper/2209.04093/) (2022) - [Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities](https://mlsanity.com/paper/2407.16243/) (2024) - [Shared Multi-modal Embedding Space for Face-Voice Association](https://mlsanity.com/paper/2512.04814/) (2025) - [Exploring Robust Face-Voice Matching in Multilingual Environments](https://mlsanity.com/paper/2407.19875/) (2024) - [Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association](https://mlsanity.com/paper/2408.02025/) (2024) - [RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2512.02860/) (2025) - [XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association](https://mlsanity.com/paper/2512.06757/) (2025) - [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026) - [POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan](https://mlsanity.com/paper/2603.24569/) (2026) - [AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification](https://mlsanity.com/paper/2606.29335/) (2026) - [More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module for Multimodal Fusion Based on Signal Theory](https://mlsanity.com/paper/2312.07212/) (2023) - [Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder](https://mlsanity.com/paper/2404.09509/) (2024) - [MuteSwap: Visual-informed Silent Video Identity Conversion](https://mlsanity.com/paper/2507.00498/) (2025) - [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025) - [SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions](https://mlsanity.com/paper/2605.11799/) (2026) - [Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge](https://mlsanity.com/paper/2607.13669/) (2026) - [Face and Voice Cross-modal Association with Learning Convex Feature Embedding](https://mlsanity.com/paper/2607.28129/) (2026) - [Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan](https://mlsanity.com/paper/2609.17913/) (2026) ## Abstract (arXiv) We study the problem of learning association between face and voice, which is gaining interest in the computer vision community lately. Prior works adopt pairwise or triplet loss formulations to learn an embedding space amenable for associated matching and verification tasks. Albeit showing some progress, such loss formulations are, however, restrictive due to dependency on distance-dependent margin parameter, poor run-time training complexity, and reliance on carefully crafted negative mining procedures. In this work, we hypothesize that enriched feature representation coupled with an effective yet efficient supervision is necessary in realizing a discriminative joint embedding space for improved face-voice association. To this end, we propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints. We coin our proposed mechanism as fusion and orthogonal projection (FOP) and instantiate in a two-stream pipeline. The overall resulting framework is evaluated on a large-scale VoxCeleb dataset with a multitude of tasks, including cross-modal verification and matching. Results show that our method performs favourably against the current state-of-the-art methods and our proposed supervision formulation is more effective and efficient than the ones employed by the contemporary methods.