# SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

arXiv: https://arxiv.org/abs/2604.25498 · HTML: https://mlsanity.com/paper/2604.25498/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [MMM : Exploring Conditional Multi-Track Music Generation with the Transformer](https://mlsanity.com/paper/2008.06048/) (2020)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer](https://mlsanity.com/paper/2111.04093/) (2021)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms](https://mlsanity.com/paper/2502.18008/) (2025)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models](https://mlsanity.com/paper/2405.09901/) (2024)
- [AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer](https://mlsanity.com/paper/2108.11213/) (2021)
- [CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages](https://mlsanity.com/paper/2502.10362/) (2025)
- [Text2midi: Generating Symbolic Music from Captions](https://mlsanity.com/paper/2412.16526/) (2024)
- [A database linking piano and orchestral MIDI scores with application to automatic projective orchestration](https://mlsanity.com/paper/1810.08611/) (2018)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset](https://mlsanity.com/paper/2410.02084/) (2024)
- [AccoMontage2: A Complete Harmonization and Accompaniment Arrangement System](https://mlsanity.com/paper/2209.00353/) (2022)
- [Q&A: Query-Based Representation Learning for Multi-Track Symbolic Music re-Arrangement](https://mlsanity.com/paper/2306.01635/) (2023)
- [METEOR: Melody-aware Texture-controllable Symbolic Orchestral Music Generation via Transformer VAE](https://mlsanity.com/paper/2409.11753/) (2024)
- [Live Orchestral Piano, a system for real-time orchestral music generation](https://mlsanity.com/paper/1609.01203/) (2016)
- [Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization](https://mlsanity.com/paper/2408.15176/) (2024)

## Built on by (most cited first, 2 of 2)

- [VIBE: Video Instruction-aligned Background music gEneration](https://mlsanity.com/paper/2608.30125/) (2026)
- [Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation](https://mlsanity.com/paper/2609.23665/) (2026)

## Abstract (arXiv)

Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration, yet existing symbolic models often struggle with a "complexity-control imbalance" between scalability and steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary orchestral generation, whose cascading decoders decompose the bar, track, and event axes, keeping decoding memory far below flat token streams and enabling conditioning at every structural level. A beat-quantized multi-pitch harmony skeleton, which may be user-written, analyzed, or model-generated, provides "short-score" conditioning, enabling outline control while producing orchestral textures. The model is refined with reinforcement learning against a cross-modal acoustic reward from CLaMP 3 audio embeddings, and a dissonance-averse sampling algorithm suppresses unintended tonal clashes during inference. Objective evaluations show that both post-training mechanisms reduce dissonance while maintaining independent melodic metrics, and in subjective tests SymphonyGen is rated above baseline systems in quality and preference, significantly so among general listeners.