# Learning to Generate Music With Sentiment

arXiv: https://arxiv.org/abs/2103.06125 · HTML: https://mlsanity.com/paper/2103.06125/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 6 of 6)

- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [Multiplicative LSTM for sequence modelling](https://mlsanity.com/paper/1609.07959/) (2016)
- [Generating Music from Literature](https://mlsanity.com/paper/1403.2124/) (2014)

## Built on by (most cited first, 29 of 29)

- [Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework](https://mlsanity.com/paper/2104.13056/) (2021)
- [Symbolic music generation conditioned on continuous-valued emotions](https://mlsanity.com/paper/2203.16165/) (2022)
- [CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models](https://mlsanity.com/paper/2410.13267/) (2024)
- [Controlling Perceived Emotion in Symbolic Music Generation with Monte Carlo Tree Search](https://mlsanity.com/paper/2208.05162/) (2022)
- [Generating music with sentiment using Transformer-GANs](https://mlsanity.com/paper/2212.11134/) (2022)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [A Survey of AI Music Generation Tools and Models](https://mlsanity.com/paper/2308.12982/) (2023)
- [Scaling Self-Supervised Representation Learning for Symbolic Piano Performance](https://mlsanity.com/paper/2506.23869/) (2025)
- [YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations](https://mlsanity.com/paper/2211.07131/) (2022)
- [Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges](https://mlsanity.com/paper/2406.08809/) (2024)
- [Towards Unified Music Emotion Recognition across Dimensional and Categorical Models](https://mlsanity.com/paper/2502.03979/) (2025)
- [REMAST: Real-time Emotion-based Music Arrangement with Soft Transition](https://mlsanity.com/paper/2305.08029/) (2023)
- [EmoGen: Eliminating Subjective Bias in Emotional Music Generation](https://mlsanity.com/paper/2307.01229/) (2023)
- [Emotion-driven Piano Music Generation via Two-stage Disentanglement and Functional Representation](https://mlsanity.com/paper/2407.20955/) (2024)
- [The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos](https://mlsanity.com/paper/2404.04420/) (2024)
- [Emotion-Driven Melody Harmonization via Melodic Variation and Functional Representation](https://mlsanity.com/paper/2407.20176/) (2024)
- [A Novel Multi-Task Learning Method for Symbolic Music Emotion Recognition](https://mlsanity.com/paper/2201.05782/) (2022)
- [Fast Diffusion GAN Model for Symbolic Music Generation Controlled by Emotions](https://mlsanity.com/paper/2310.14040/) (2023)
- [Emotion-Conditioned Melody Harmonization with Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2306.03718/) (2023)
- [Emotion4MIDI: a Lyrics-based Emotion-Labeled Symbolic Music Dataset](https://mlsanity.com/paper/2307.14783/) (2023)
- [PIAST: A Multimodal Piano Dataset with Audio, Symbolic and Text](https://mlsanity.com/paper/2411.02551/) (2024)
- [Music Sentiment Transfer](https://mlsanity.com/paper/2110.05765/) (2021)
- [FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control](https://mlsanity.com/paper/2201.10936/) (2022)
- [EMelodyGen: Emotion-Conditioned Melody Generation in ABC Notation with the Musical Feature Template](https://mlsanity.com/paper/2309.13259/) (2023)
- [Agent-Driven Large Language Models for Mandarin Lyric Generation](https://mlsanity.com/paper/2410.01450/) (2024)
- [Emotion Manipulation Through Music -- A Deep Learning Interactive Visual Approach](https://mlsanity.com/paper/2406.08623/) (2024)
- [AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation](https://mlsanity.com/paper/2509.00813/) (2025)
- [Story2MIDI: Emotionally Aligned Music Generation from Text](https://mlsanity.com/paper/2512.02192/) (2025)
- [Memo2496: Expert-Annotated Dataset and Dual-view Adaptive Framework for Music Emotion Recognition](https://mlsanity.com/paper/2512.13998/) (2025)

## Abstract (arXiv)

Deep Learning models have shown very promising results in automatically composing polyphonic music pieces. However, it is very hard to control such models in order to guide the compositions towards a desired goal. We are interested in controlling a model to automatically generate music with a given sentiment. This paper presents a generative Deep Learning model that can be directed to compose music with a given sentiment. Besides music generation, the same model can be used for sentiment analysis of symbolic music. We evaluate the accuracy of the model in classifying sentiment of symbolic music using a new dataset of video game soundtracks. Results show that our model is able to obtain good prediction accuracy. A user study shows that human subjects agreed that the generated music has the intended sentiment, however negative pieces can be ambiguous.