# A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios

arXiv: https://arxiv.org/abs/2409.17864 · HTML: https://mlsanity.com/paper/2409.17864/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 88 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Mining Latent Structures for Multimedia Recommendation](https://mlsanity.com/paper/2104.09036/) (2021)
- [Contrastive Learning for Cold-Start Recommendation](https://mlsanity.com/paper/2107.05315/) (2021)
- [Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences](https://mlsanity.com/paper/2307.14225/) (2023)
- [Seeing Voices and Hearing Faces: Cross-modal biometric matching](https://mlsanity.com/paper/1804.00326/) (2018)
- [Learnable PINs: Cross-Modal Embeddings for Person Identity](https://mlsanity.com/paper/1805.00833/) (2018)
- [musicnn: Pre-trained convolutional neural networks for music audio tagging](https://mlsanity.com/paper/1909.06654/) (2019)
- [Current Challenges and Visions in Music Recommender Systems Research](https://mlsanity.com/paper/1710.03208/) (2017)
- [Fusion and Orthogonal Projection for Improved Face-Voice Association](https://mlsanity.com/paper/2112.10483/) (2021)
- [End-to-end learning for music audio tagging at scale](https://mlsanity.com/paper/1711.02520/) (2017)
- [Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition](https://mlsanity.com/paper/1904.01356/) (2019)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendation](https://mlsanity.com/paper/2306.14462/) (2023)
- [An Unified Search and Recommendation Foundation Model for Cold-Start Scenario](https://mlsanity.com/paper/2309.08939/) (2023)

## Built on by (most cited first, 13 of 13)

- [Just Ask for Music (JAM): Multimodal and Personalized Natural Language Music Recommendation](https://mlsanity.com/paper/2507.15826/) (2025)
- [Disentangling and Generating Modalities for Recommendation in Missing Modality Scenarios](https://mlsanity.com/paper/2504.16352/) (2025)
- [NTSFormer: A Self-Teaching Graph Transformer for Multimodal Isolated Cold-Start Node Classification](https://mlsanity.com/paper/2507.04870/) (2025)
- [Warmer for Less: A Cost-Efficient Strategy for Cold-Start Recommendations at Pinterest](https://mlsanity.com/paper/2512.17277/) (2025)
- [Multi-modal Adaptive Mixture of Experts for Cold-start Recommendation](https://mlsanity.com/paper/2508.08042/) (2025)
- [POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan](https://mlsanity.com/paper/2603.24569/) (2026)
- [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025)
- [RobustA: Robust Anomaly Detection in Multimodal Data](https://mlsanity.com/paper/2511.07276/) (2025)
- [SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions](https://mlsanity.com/paper/2605.11799/) (2026)
- [Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge](https://mlsanity.com/paper/2607.13669/) (2026)
- [HyperAgent4POI: Dynamic Semantic Message Passing on Multi-Agent Hypergraphs for Missing-Modality Recommendation](https://mlsanity.com/paper/2608.01846/) (2026)
- [Sequential Modality Dropout for Robust Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2608.10240/) (2026)
- [Bridging the Topological Divide: Dual-Branch Graph Distillation with Semantic Priors for Robust Cold-Start Recommendation](https://mlsanity.com/paper/2610.19578/) (2026)

## Abstract (arXiv)

Most recommender systems adopt collaborative filtering (CF) and provide recommendations based on past collective interactions. Therefore, the performance of CF algorithms degrades when few or no interactions are available, a scenario referred to as cold-start. To address this issue, previous work relies on models leveraging both collaborative data and side information on the users or items. Similar to multimodal learning, these models aim at combining collaborative and content representations in a shared embedding space. In this work we propose a novel technique for multimodal recommendation, relying on a multimodal Single-Branch embedding network for Recommendation (SiBraR). Leveraging weight-sharing, SiBraR encodes interaction data as well as multimodal side information using the same single-branch embedding network on different modalities. This makes SiBraR effective in scenarios of missing modality, including cold start. Our extensive experiments on large-scale recommendation datasets from three different recommendation domains (music, movie, and e-commerce) and providing multimodal content information (audio, text, image, labels, and interactions) show that SiBraR significantly outperforms CF as well as state-of-the-art content-based RSs in cold-start scenarios, and is competitive in warm scenarios. We show that SiBraR's recommendations are accurate in missing modality scenarios, and that the model is able to map different modalities to the same region of the shared embedding space, hence reducing the modality gap.