# Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition

arXiv: https://arxiv.org/abs/2104.06517 · HTML: https://mlsanity.com/paper/2104.06517/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification](https://mlsanity.com/paper/1608.04363/) (2016)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Transfer learning for music classification and regression tasks](https://mlsanity.com/paper/1703.09179/) (2017)
- [End-to-End Environmental Sound Classification using a 1D Convolutional Neural Network](https://mlsanity.com/paper/1904.08990/) (2019)
- [Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms](https://mlsanity.com/paper/1710.10451/) (2017)
- [Regression-based music emotion prediction using triplet neural networks](https://mlsanity.com/paper/2001.09988/) (2020)
- [Multimodal Deep Models for Predicting Affective Responses Evoked by Movies](https://mlsanity.com/paper/1909.06957/) (2019)
- [Randomly weighted CNNs for (music) audio classification](https://mlsanity.com/paper/1805.00237/) (2018)
- [A Siamese Neural Network with Modified Distance Loss For Transfer Learning in Speech Emotion Recognition](https://mlsanity.com/paper/2006.03001/) (2020)
- [Enhancing Music Features by Knowledge Transfer from User-item Log Data](https://mlsanity.com/paper/1903.02794/) (2019)

## Built on by (most cited first, 15 of 15)

- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming](https://mlsanity.com/paper/2211.01317/) (2022)
- [ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction](https://mlsanity.com/paper/2404.18470/) (2024)
- [Audio Embeddings as Teachers for Music Classification](https://mlsanity.com/paper/2306.17424/) (2023)
- [YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls](https://mlsanity.com/paper/2412.09168/) (2024)
- [Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?](https://mlsanity.com/paper/2506.01482/) (2025)
- [A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2304.11472/) (2023)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Music auto-tagging in the long tail: A few-shot approach](https://mlsanity.com/paper/2409.07730/) (2024)
- [Emotion Manipulation Through Music -- A Deep Learning Interactive Visual Approach](https://mlsanity.com/paper/2406.08623/) (2024)
- [Semi-Supervised Self-Learning Enhanced Music Emotion Recognition](https://mlsanity.com/paper/2410.21897/) (2024)
- [A Study on the Data Distribution Gap in Music Emotion Recognition](https://mlsanity.com/paper/2510.04688/) (2025)
- [On the Emotion Understanding of Synthesized Speech](https://mlsanity.com/paper/2603.16483/) (2026)

## Abstract (arXiv)

Emotion is a complicated notion present in music that is hard to capture even with fine-tuned feature engineering. In this paper, we investigate the utility of state-of-the-art pre-trained deep audio embedding methods to be used in the Music Emotion Recognition (MER) task. Deep audio embedding methods allow us to efficiently capture the high dimensional features into a compact representation. We implement several multi-class classifiers with deep audio embeddings to predict emotion semantics in music. We investigate the effectiveness of L3-Net and VGGish deep audio embedding methods for music emotion inference over four music datasets. The experiments with several classifiers on the task show that the deep audio embedding solutions can improve the performances of the previous baseline MER models. We conclude that deep audio embeddings represent musical emotion semantics for the MER task without expert human engineering.