# Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation

arXiv: https://arxiv.org/abs/2107.07268 · HTML: https://mlsanity.com/paper/2107.07268/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2021.

## Built on by (most cited first, 10 of 10)

- [MuseChat: A Conversational Music Recommendation System for Videos](https://mlsanity.com/paper/2310.06282/) (2023)
- [HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization](https://mlsanity.com/paper/2503.01725/) (2025)
- [FilmComposer: LLM-Driven Music Production for Silent Film Clips](https://mlsanity.com/paper/2503.08147/) (2025)
- [CPGRec+: A Balance-oriented Framework for Personalized Video Game Recommendations](https://mlsanity.com/paper/2604.14586/) (2026)
- [Human Pose Driven Object Effects Recommendation](https://mlsanity.com/paper/2209.08353/) (2022)
- [Chorus: Harmonizing Context and Sensing Signals for Data-Free Model Customization in IoT](https://mlsanity.com/paper/2512.15206/) (2025)
- [Variational Adapter for Cross-modal Similarity Representation](https://mlsanity.com/paper/2605.30968/) (2026)
- [VMCML: Video and Music Matching via Cross-Modality Lifting](https://mlsanity.com/paper/2303.12379/) (2023)
- [Solving Copyright Infringement on Short Video Platforms: Novel Datasets and an Audio Restoration Deep Learning Pipeline](https://mlsanity.com/paper/2504.21772/) (2025)
- [Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs](https://mlsanity.com/paper/2604.08846/) (2026)

## Abstract (arXiv)

In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.