# Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

arXiv: https://arxiv.org/abs/2009.05103 · HTML: https://mlsanity.com/paper/2009.05103/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [Computational Emotion Analysis From Images: Recent Advances and Future Directions](https://mlsanity.com/paper/2103.10798/) (2021)
- [A Multimodal Symphony: Integrating Taste and Sound through Generative AI](https://mlsanity.com/paper/2503.02823/) (2025)
- [Emotion-Guided Image to Music Generation](https://mlsanity.com/paper/2410.22299/) (2024)
- [MMVA: Multimodal Matching Based on Valence and Arousal across Images, Music, and Musical Captions](https://mlsanity.com/paper/2501.01094/) (2025)
- [MPJudge: Towards Perceptual Assessment of Music-Induced Paintings](https://mlsanity.com/paper/2511.07137/) (2025)
- [Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment](https://mlsanity.com/paper/2602.17599/) (2026)

## Abstract (arXiv)

Both images and music can convey rich semantics and are widely used to induce specific emotions. Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. Existing emotion-based image and music matching methods either employ limited categorical emotion states which cannot well reflect the complexity and subtlety of emotions, or train the matching model using an impractical multi-stage pipeline. In this paper, we study end-to-end matching between image and music based on emotions in the continuous valence-arousal (VA) space. First, we construct a large-scale dataset, termed Image-Music-Emotion-Matching-Net (IMEMNet), with over 140K image-music pairs. Second, we propose cross-modal deep continuous metric learning (CDCML) to learn a shared latent embedding space which preserves the cross-modal similarity relationship in the continuous matching space. Finally, we refine the embedding space by further preserving the single-modal emotion relationship in the VA spaces of both images and music. The metric learning in the embedding space and task regression in the label space are jointly optimized for both cross-modal matching and single-modal VA prediction. The extensive experiments conducted on IMEMNet demonstrate the superiority of CDCML for emotion-based image and music matching as compared to the state-of-the-art approaches.