# Single-branch Network for Multimodal Training

arXiv: https://arxiv.org/abs/2303.06129 · HTML: https://mlsanity.com/paper/2303.06129/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [On Learning Associations of Faces and Voices](https://mlsanity.com/paper/1805.05553/) (2018)
- [Cross-modal Speaker Verification and Recognition: A Multilingual Perspective](https://mlsanity.com/paper/2004.13780/) (2020)
- [Audio-visual Speaker Recognition with a Cross-modal Discriminative Network](https://mlsanity.com/paper/2008.03894/) (2020)
- [A Multi-View Approach To Audio-Visual Speaker Verification](https://mlsanity.com/paper/2102.06291/) (2021)
- [Disjoint Mapping Network for Cross-modal Matching of Voices and Faces](https://mlsanity.com/paper/1807.04836/) (2018)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Deep Latent Space Learning for Cross-modal Mapping of Audio and Visual Signals](https://mlsanity.com/paper/1909.08685/) (2019)
- [Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition](https://mlsanity.com/paper/1904.01356/) (2019)
- [Image and Encoded Text Fusion for Multi-Modal Classification](https://mlsanity.com/paper/1810.02001/) (2018)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [Git Loss for Deep Face Recognition](https://mlsanity.com/paper/1807.08512/) (2018)
- [Semantically Grounded Visual Embeddings for Zero-Shot Learning](https://mlsanity.com/paper/2201.00577/) (2022)
- [Learning Branched Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2208.10238/) (2022)
- [Guiding Attention using Partial-Order Relationships for Image Captioning](https://mlsanity.com/paper/2204.07476/) (2022)

## Built on by (most cited first, 13 of 13)

- [A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios](https://mlsanity.com/paper/2409.17864/) (2024)
- [PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association](https://mlsanity.com/paper/2505.17002/) (2025)
- [Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challenge](https://mlsanity.com/paper/2512.20376/) (2025)
- [Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan](https://mlsanity.com/paper/2404.09342/) (2024)
- [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024)
- [DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement Estimation in Conversation](https://mlsanity.com/paper/2308.01966/) (2023)
- [Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association](https://mlsanity.com/paper/2408.02025/) (2024)
- [RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association](https://mlsanity.com/paper/2512.02860/) (2025)
- [XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association](https://mlsanity.com/paper/2512.06757/) (2025)
- [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026)
- [AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification](https://mlsanity.com/paper/2606.29335/) (2026)
- [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025)
- [Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan](https://mlsanity.com/paper/2609.17913/) (2026)

## Abstract (arXiv)

With the rapid growth of social media platforms, users are sharing billions of multimedia posts containing audio, images, and text. Researchers have focused on building autonomous systems capable of processing such multimedia data to solve challenging multimodal tasks including cross-modal retrieval, matching, and verification. Existing works use separate networks to extract embeddings of each modality to bridge the gap between them. The modular structure of their branched networks is fundamental in creating numerous multimodal applications and has become a defacto standard to handle multiple modalities. In contrast, we propose a novel single-branch network capable of learning discriminative representation of unimodal as well as multimodal tasks without changing the network. An important feature of our single-branch network is that it can be trained either using single or multiple modalities without sacrificing performance. We evaluated our proposed single-branch network on the challenging multimodal problem (face-voice association) for cross-modal verification and matching tasks with various loss formulations. Experimental results demonstrate the superiority of our proposed single-branch network over the existing methods in a wide range of experiments. Code: https://github.com/msaadsaeed/SBNet