# Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals

arXiv: https://arxiv.org/abs/1909.08685 · HTML: https://mlsanity.com/paper/1909.08685/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association](https://mlsanity.com/paper/2505.17002/) (2025)
- [Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plan](https://mlsanity.com/paper/2508.04592/) (2025)
- [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025)
- [Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan](https://mlsanity.com/paper/2404.09342/) (2024)
- [Learning Branched Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2208.10238/) (2022)
- [A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition](https://mlsanity.com/paper/2210.10972/) (2022)
- [Exploring Robust Face-Voice Matching in Multilingual Environments](https://mlsanity.com/paper/2407.19875/) (2024)
- [RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2512.02860/) (2025)
- [XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association](https://mlsanity.com/paper/2512.06757/) (2025)
- [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026)
- [AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification](https://mlsanity.com/paper/2606.29335/) (2026)
- [Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder](https://mlsanity.com/paper/2404.09509/) (2024)
- [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025)
- [Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration](https://mlsanity.com/paper/2606.03802/) (2026)
- [Face and Voice Cross-modal Association with Learning Convex Feature Embedding](https://mlsanity.com/paper/2607.28129/) (2026)
- [Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan](https://mlsanity.com/paper/2609.17913/) (2026)

## Abstract (arXiv)

We propose a novel deep training algorithm for joint representation of audio and visual information which consists of a single stream network (SSNet) coupled with a novel loss function to learn a shared deep latent space representation of multimodal information. The proposed framework characterizes the shared latent space by leveraging the class centers which helps to eliminate the need for pairwise or triplet supervision. We quantitatively and qualitatively evaluate the proposed approach on VoxCeleb, a benchmarks audio-visual dataset on a multitude of tasks including cross-modal verification, cross-modal matching, and cross-modal retrieval. State-of-the-art performance is achieved on cross-modal verification and matching while comparable results are observed on the remaining applications. Our experiments demonstrate the effectiveness of the technique for cross-modal biometric applications.