# Story2MIDI: Emotionally Aligned Music Generation from Text

arXiv: https://arxiv.org/abs/2512.02192 · HTML: https://mlsanity.com/paper/2512.02192/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [The emotional arcs of stories are dominated by six basic shapes](https://mlsanity.com/paper/1606.07772/) (2016)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [Modeling Protagonist Emotions for Emotion-Aware Storytelling](https://mlsanity.com/paper/2010.06822/) (2020)
- [GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music](https://mlsanity.com/paper/2010.07061/) (2020)
- [Learning to Generate Music With Sentiment](https://mlsanity.com/paper/2103.06125/) (2021)
- [Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions](https://mlsanity.com/paper/2403.01222/) (2024)
- [Computer-Generated Music for Tabletop Role-Playing Games](https://mlsanity.com/paper/2008.07009/) (2020)
- [Text2midi: Generating Symbolic Music from Captions](https://mlsanity.com/paper/2412.16526/) (2024)
- [Evaluating generative audio systems and their metrics](https://mlsanity.com/paper/2209.00130/) (2022)
- [The Emotion Dynamics of Literary Novels](https://mlsanity.com/paper/2403.02474/) (2024)
- [AI-Based Affective Music Generation Systems: A Review of Methods, and Challenges](https://mlsanity.com/paper/2301.06890/) (2023)
- [Adaptive Music Composition for Games](https://mlsanity.com/paper/1907.01154/) (2019)
- [Are Words Enough? On the semantic conditioning of affective music generation](https://mlsanity.com/paper/2311.03624/) (2023)
- [Neural association between musical features and shared emotional perception while movie-watching: fMRI study](https://mlsanity.com/paper/2410.17300/) (2024)

## Abstract (arXiv)

In this paper, we introduce Story2MIDI, a sequence-to-sequence Transformer-based model for generating emotion-aligned music from a given piece of text. To develop this model, we construct the Story2MIDI dataset by merging existing datasets for sentiment analysis from text and emotion classification in music. The resulting dataset contains pairs of text blurbs and music pieces that evoke the same emotions in the reader or listener. Despite the small scale of our dataset and limited computational resources, our results indicate that our model effectively learns emotion-relevant features in music and incorporates them into its generation process, producing samples with diverse emotional responses. We evaluate the generated outputs using objective musical metrics and a human listening study, confirming the model's ability to capture intended emotional cues.