# REMAST: Real-time Emotion-based Music Arrangement with Soft Transition

arXiv: https://arxiv.org/abs/2305.08029 · HTML: https://mlsanity.com/paper/2305.08029/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 65 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 11 of 11)

- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [A Deep Journey into Super-resolution: A survey](https://mlsanity.com/paper/1904.07523/) (2019)
- [EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generation](https://mlsanity.com/paper/2108.01374/) (2021)
- [Learning to Generate Music With Sentiment](https://mlsanity.com/paper/2103.06125/) (2021)
- [Recurrent Topic-Transition GAN for Visual Paragraph Generation](https://mlsanity.com/paper/1703.07022/) (2017)
- [Automatic Melody Harmonization with Triad Chords: A Comparative Study](https://mlsanity.com/paper/2001.02360/) (2020)
- [RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning](https://mlsanity.com/paper/2002.03082/) (2020)
- [Symbolic music generation conditioned on continuous-valued emotions](https://mlsanity.com/paper/2203.16165/) (2022)
- [SongDriver: Real-time Music Accompaniment Generation without Logical Latency nor Exposure Bias](https://mlsanity.com/paper/2209.06054/) (2022)
- [Adaptive Music Composition for Games](https://mlsanity.com/paper/1907.01154/) (2019)
- [A Comparative Study of Western and Chinese Classical Music based on Soundscape Models](https://mlsanity.com/paper/2002.09021/) (2020)

## Built on by (most cited first, 5 of 5)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music](https://mlsanity.com/paper/2402.09871/) (2024)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Multi-Accent Mandarin Dry-Vocal Singing Dataset: Benchmark for Singing Accent Recognition](https://mlsanity.com/paper/2512.07005/) (2025)
- [Singing Timbre Popularity Assessment Based on Multimodal Large Foundation Model](https://mlsanity.com/paper/2512.06999/) (2025)

## Abstract (arXiv)

Music as an emotional intervention medium has important applications in scenarios such as music therapy, games, and movies. However, music needs real-time arrangement according to changing emotions, bringing challenges to balance emotion real-time fit and soft emotion transition due to the fine-grained and mutable nature of the target emotion. Existing studies mainly focus on achieving emotion real-time fit, while the issue of smooth transition remains understudied, affecting the overall emotional coherence of the music. In this paper, we propose REMAST to address this trade-off. Specifically, we recognize the last timestep's music emotion and fuse it with the current timestep's input emotion. The fused emotion then guides REMAST to generate the music based on the input melody. To adjust music similarity and emotion real-time fit flexibly, we downsample the original melody and feed it into the generation model. Furthermore, we design four music theory features by domain knowledge to enhance emotion information and employ semi-supervised learning to mitigate the subjective bias introduced by manual dataset annotation. According to the evaluation results, REMAST surpasses the state-of-the-art methods in objective and subjective metrics. These results demonstrate that REMAST achieves real-time fit and smooth transition simultaneously, enhancing the coherence of the generated music.