# PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association

arXiv: https://arxiv.org/abs/2505.17002 · HTML: https://mlsanity.com/paper/2505.17002/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Disjoint Mapping Network for Cross-modal Matching of Voices and Faces](https://mlsanity.com/paper/1807.04836/) (2018)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals](https://mlsanity.com/paper/1909.08685/) (2019)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement Estimation in Conversation](https://mlsanity.com/paper/2308.01966/) (2023)
- [Guiding Attention using Partial-Order Relationships for Image Captioning](https://mlsanity.com/paper/2204.07476/) (2022)

## Built on by (most cited first, 9 of 9)

- [Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plan](https://mlsanity.com/paper/2508.04592/) (2025)
- [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025)
- [RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2512.02860/) (2025)
- [XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association](https://mlsanity.com/paper/2512.06757/) (2025)
- [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026)
- [AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification](https://mlsanity.com/paper/2606.29335/) (2026)
- [MuteSwap: Visual-informed Silent Video Identity Conversion](https://mlsanity.com/paper/2507.00498/) (2025)
- [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025)
- [Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan](https://mlsanity.com/paper/2609.17913/) (2026)

## Abstract (arXiv)

We study the task of learning association between faces and voices, which is gaining interest in the multimodal community lately. These methods suffer from the deliberate crafting of negative mining procedures as well as the reliance on the distant margin parameter. These issues are addressed by learning a joint embedding space in which orthogonality constraints are applied to the fused embeddings of faces and voices. However, embedding spaces of faces and voices possess different characteristics and require spaces to be aligned before fusing them. To this end, we propose a method that accurately aligns the embedding spaces and fuses them with an enhanced gated fusion thereby improving the performance of face-voice association. Extensive experiments on the VoxCeleb dataset reveals the merits of the proposed approach.