# Generating music with sentiment using Transformer-GANs

arXiv: https://arxiv.org/abs/2212.11134 · HTML: https://mlsanity.com/paper/2212.11134/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution](https://mlsanity.com/paper/1611.04051/) (2016)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [MusPy: A Toolkit for Symbolic Music Generation](https://mlsanity.com/paper/2008.01951/) (2020)
- [Learning to Generate Music With Sentiment](https://mlsanity.com/paper/2103.06125/) (2021)
- [Computer-Generated Music for Tabletop Role-Playing Games](https://mlsanity.com/paper/2008.07009/) (2020)

## Built on by (most cited first, 11 of 11)

- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [EmoGen: Eliminating Subjective Bias in Emotional Music Generation](https://mlsanity.com/paper/2307.01229/) (2023)
- [Emotion-driven Piano Music Generation via Two-stage Disentanglement and Functional Representation](https://mlsanity.com/paper/2407.20955/) (2024)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation](https://mlsanity.com/paper/2304.08953/) (2023)
- [MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion](https://mlsanity.com/paper/2312.10307/) (2023)
- [Diffusion-based Symbolic Music Generation with Structured State Space Models](https://mlsanity.com/paper/2507.20128/) (2025)
- [MMT-BERT: Chord-aware Symbolic Music Generation Based on Multitrack Music Transformer and MusicBERT](https://mlsanity.com/paper/2409.00919/) (2024)

## Abstract (arXiv)

The field of Automatic Music Generation has seen significant progress thanks to the advent of Deep Learning. However, most of these results have been produced by unconditional models, which lack the ability to interact with their users, not allowing them to guide the generative process in meaningful and practical ways. Moreover, synthesizing music that remains coherent across longer timescales while still capturing the local aspects that make it sound ``realistic'' or ``human-like'' is still challenging. This is due to the large computational requirements needed to work with long sequences of data, and also to limitations imposed by the training schemes that are often employed. In this paper, we propose a generative model of symbolic music conditioned by data retrieved from human sentiment. The model is a Transformer-GAN trained with labels that correspond to different configurations of the valence and arousal dimensions that quantitatively represent human affective states. We try to tackle both of the problems above by employing an efficient linear version of Attention and using a Discriminator both as a tool to improve the overall quality of the generated music and its ability to follow the conditioning signals.