# Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges

arXiv: https://arxiv.org/abs/2406.08809 · HTML: https://mlsanity.com/paper/2406.08809/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 158 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [NGC: A Unified Framework for Learning with Open-World Noisy Data](https://mlsanity.com/paper/2108.11035/) (2021)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [Learning to Generate Music With Sentiment](https://mlsanity.com/paper/2103.06125/) (2021)
- [MorpheuS: generating structured music with constrained patterns and tension](https://mlsanity.com/paper/1812.04832/) (2018)
- [Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework](https://mlsanity.com/paper/2104.13056/) (2021)
- [Music Mood Detection Based On Audio And Lyrics With Deep Neural Net](https://mlsanity.com/paper/1809.07276/) (2018)
- [Regression-based music emotion prediction using triplet neural networks](https://mlsanity.com/paper/2001.09988/) (2020)
- [Tracing Back Music Emotion Predictions to Sound Sources and Intuitive Perceptual Qualities](https://mlsanity.com/paper/2106.07787/) (2021)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [Transformer-based approach towards music emotion recognition from lyrics](https://mlsanity.com/paper/2101.02051/) (2021)
- [YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations](https://mlsanity.com/paper/2211.07131/) (2022)
- [Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses](https://mlsanity.com/paper/2202.10453/) (2022)
- [Efficient Supervised Training of Audio Transformers for Music Representation Learning](https://mlsanity.com/paper/2309.16418/) (2023)
- [AttendAffectNet: Self-Attention based Networks for Predicting Affective Responses from Movies](https://mlsanity.com/paper/2010.11188/) (2020)
- [Cross-cultural Mood Perception in Pop Songs and its Alignment with Mood Detection Algorithms](https://mlsanity.com/paper/2108.00768/) (2021)
- [A Comparative Study of Western and Chinese Classical Music based on Soundscape Models](https://mlsanity.com/paper/2002.09021/) (2020)
- [Semi-supervised music emotion recognition using noisy student training and harmonic pitch class profiles](https://mlsanity.com/paper/2112.00702/) (2021)
- [A Novel Multi-Task Learning Method for Symbolic Music Emotion Recognition](https://mlsanity.com/paper/2201.05782/) (2022)
- [An empirical approach to the relationship between emotion and music production quality](https://mlsanity.com/paper/1803.11154/) (2018)
- [Modelling Emotion Dynamics in Song Lyrics with State Space Models](https://mlsanity.com/paper/2210.09434/) (2022)

## Built on by (most cited first, 7 of 7)

- [Towards Unified Music Emotion Recognition across Dimensional and Categorical Models](https://mlsanity.com/paper/2502.03979/) (2025)
- [Leveraging LLM Embeddings for Cross Dataset Label Alignment and Zero Shot Music Emotion Prediction](https://mlsanity.com/paper/2410.11522/) (2024)
- [Taste-aware music retrieval from audio embeddings](https://mlsanity.com/paper/2607.03296/) (2026)
- [A Study on the Data Distribution Gap in Music Emotion Recognition](https://mlsanity.com/paper/2510.04688/) (2025)
- [Words to Waves: Emotion-Adaptive Music Recommendation System](https://mlsanity.com/paper/2510.21724/) (2025)
- [Memo2496: Expert-Annotated Dataset and Dual-view Adaptive Framework for Music Emotion Recognition](https://mlsanity.com/paper/2512.13998/) (2025)
- [DialBGM: A Benchmark for Background Music Recommendation from Everyday Multi-Turn Dialogues](https://mlsanity.com/paper/2604.07895/) (2026)

## Abstract (arXiv)

Deep learning models for music have advanced drastically in recent years, but how good are machine learning models at capturing emotion, and what challenges are researchers facing? In this paper, we provide a comprehensive overview of the available music-emotion datasets and discuss evaluation standards as well as competitions in the field. We also offer a brief overview of various types of music emotion prediction models that have been built over the years, providing insights into the diverse approaches within the field. Through this examination, we highlight the challenges that persist in accurately capturing emotion in music, including issues related to dataset quality, annotation consistency, and model generalization. Additionally, we explore the impact of different modalities, such as audio, MIDI, and physiological signals, on the effectiveness of emotion prediction models. Through this examination, we identify persistent challenges in music emotion recognition (MER), including issues related to dataset quality, the ambiguity in emotion labels, and the difficulties of cross-dataset generalization. We argue that future advancements in MER require standardized benchmarks, larger and more diverse datasets, and improved model interpretability. Recognizing the dynamic nature of this field, we have complemented our findings with an accompanying GitHub repository. This repository contains a comprehensive list of music emotion datasets and recent predictive models.