# Emotion-Conditioned Melody Harmonization with Hierarchical Variational Autoencoder

arXiv: https://arxiv.org/abs/2306.03718 · HTML: https://mlsanity.com/paper/2306.03718/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions](https://mlsanity.com/paper/2011.06801/) (2020)
- [Learning to Generate Music With Sentiment](https://mlsanity.com/paper/2103.06125/) (2021)
- [Computer-Generated Music for Tabletop Role-Playing Games](https://mlsanity.com/paper/2008.07009/) (2020)
- [Automatic Melody Harmonization with Triad Chords: A Comparative Study](https://mlsanity.com/paper/2001.02360/) (2020)
- [Generating Lead Sheets with Affect: A Novel Conditional seq2seq Framework](https://mlsanity.com/paper/2104.13056/) (2021)
- [Generating Thematic Chinese Poetry using Conditional Variational Autoencoders with Hybrid Decoders](https://mlsanity.com/paper/1711.07632/) (2017)
- [Controlling Perceived Emotion in Symbolic Music Generation with Monte Carlo Tree Search](https://mlsanity.com/paper/2208.05162/) (2022)
- [SurpriseNet: Melody Harmonization Conditioning on User-controlled Surprise Contours](https://mlsanity.com/paper/2108.00378/) (2021)
- [Chord Generation from Symbolic Melody Using BLSTM Networks](https://mlsanity.com/paper/1712.01011/) (2017)
- [Melody Harmonization Using Orderless NADE, Chord Balancing, and Blocked Gibbs Sampling](https://mlsanity.com/paper/2010.13468/) (2020)

## Built on by (most cited first, 2 of 2)

- [Emotion-Driven Melody Harmonization via Melodic Variation and Functional Representation](https://mlsanity.com/paper/2407.20176/) (2024)
- [Incorporating Structure and Chord Constraints in Symbolic Transformer-based Melodic Harmonization](https://mlsanity.com/paper/2512.07627/) (2025)

## Abstract (arXiv)

Existing melody harmonization models have made great progress in improving the quality of generated harmonies, but most of them ignored the emotions beneath the music. Meanwhile, the variability of harmonies generated by previous methods is insufficient. To solve these problems, we propose a novel LSTM-based Hierarchical Variational Auto-Encoder (LHVAE) to investigate the influence of emotional conditions on melody harmonization, while improving the quality of generated harmonies and capturing the abundant variability of chord progressions. Specifically, LHVAE incorporates latent variables and emotional conditions at different levels (piece- and bar-level) to model the global and local music properties. Additionally, we introduce an attention-based melody context vector at each step to better learn the correspondence between melodies and harmonies. Objective experimental results show that our proposed model outperforms other LSTM-based models. Through subjective evaluation, we conclude that only altering the types of chords hardly changes the overall emotion of the music. The qualitative analysis demonstrates the ability of our model to generate variable harmonies.