# Video-to-Music Recommendation using Temporal Alignment of Segments

arXiv: https://arxiv.org/abs/2306.07187 · HTML: https://mlsanity.com/paper/2306.07187/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval](https://mlsanity.com/paper/2008.10238/) (2020)
- [Cross-modal Embeddings for Video and Audio Retrieval](https://mlsanity.com/paper/1801.02200/) (2018)
- [Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA](https://mlsanity.com/paper/1908.03744/) (2019)
- [TransNet: A deep network for fast detection of common shot transitions](https://mlsanity.com/paper/1906.03363/) (2019)
- [Curriculum Audiovisual Learning](https://mlsanity.com/paper/2001.09414/) (2020)
- [AlignNet: A Unifying Approach to Audio-Visual Alignment](https://mlsanity.com/paper/2002.05070/) (2020)
- [Cross-Modal Music-Video Recommendation: A Study of Design Choices](https://mlsanity.com/paper/2104.14799/) (2021)
- [Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval](https://mlsanity.com/paper/1711.08976/) (2017)
- [Is there a "language of music-video clips" ? A qualitative and quantitative study](https://mlsanity.com/paper/2108.00970/) (2021)
- [Representation Learning for Image-based Music Recommendation](https://mlsanity.com/paper/1808.09198/) (2018)
- [Multi-Modal Music Information Retrieval: Augmenting Audio-Analysis with Visual Computing for Improved Music Video Analysis](https://mlsanity.com/paper/2002.00251/) (2020)

## Built on by (most cited first, 1 of 1)

- [CPGRec+: A Balance-oriented Framework for Personalized Video Game Recommendations](https://mlsanity.com/paper/2604.14586/) (2026)

## Abstract (arXiv)

We study cross-modal recommendation of music tracks to be used as soundtracks for videos. This problem is known as the music supervision task. We build on a self-supervised system that learns a content association between music and video. In addition to the adequacy of content, adequacy of structure is crucial in music supervision to obtain relevant recommendations. We propose a novel approach to significantly improve the system's performance using structure-aware recommendation. The core idea is to consider not only the full audio-video clips, but rather shorter segments for training and inference. We find that using semantic segments and ranking the tracks according to sequence alignment costs significantly improves the results. We investigate the impact of different ranking metrics and segmentation methods.