# AnyAccomp: Generalizable Accompaniment Generation via Quantized Melodic Bottleneck

arXiv: https://arxiv.org/abs/2509.14052 · HTML: https://mlsanity.com/paper/2509.14052/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [YuE: Scaling Open Foundation Models for Long-Form Music Generation](https://mlsanity.com/paper/2503.08638/) (2025)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [SingSong: Generating musical accompaniments from singing](https://mlsanity.com/paper/2301.12662/) (2023)
- [Moisesdb: A dataset for source separation beyond 4-stems](https://mlsanity.com/paper/2307.15913/) (2023)
- [Music Source Separation with Band-Split RoPE Transformer](https://mlsanity.com/paper/2309.02612/) (2023)
- [Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement](https://mlsanity.com/paper/2502.07243/) (2025)
- [Amphion: An Open-Source Audio, Music and Speech Generation Toolkit](https://mlsanity.com/paper/2312.09911/) (2023)
- [FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation](https://mlsanity.com/paper/2405.07682/) (2024)
- [Overview of the Amphion Toolkit (v0.2)](https://mlsanity.com/paper/2501.15442/) (2025)
- [SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset](https://mlsanity.com/paper/2505.09325/) (2025)
- [Accompaniment Prompt Adherence: A Measure for Evaluating Music Accompaniment Systems](https://mlsanity.com/paper/2503.06346/) (2025)

## Built on by (most cited first, 2 of 2)

- [Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance](https://mlsanity.com/paper/2607.11143/) (2026)
- [Iterative Audio Separation with Mixture Consistency via MIMO Model Extension](https://mlsanity.com/paper/2609.07226/) (2026)

## Abstract (arXiv)

Singing Accompaniment Generation (SAG) is the process of generating instrumental music for a given clean vocal input. However, existing SAG techniques use source-separated vocals as input and overfit to separation artifacts. This creates a critical train-test mismatch, leading to failure on clean, real-world vocal inputs. We introduce AnyAccomp, a framework that resolves this by decoupling accompaniment generation from source-dependent artifacts. AnyAccomp first employs a quantized melodic bottleneck, using a chromagram and a VQ-VAE to extract a discrete and timbre-invariant representation of the core melody. A subsequent flow-matching model then generates the accompaniment conditioned on these robust codes. Experiments show AnyAccomp achieves competitive performance on separated-vocal benchmarks while significantly outperforming baselines on generalization test sets of clean studio vocals and, notably, solo instrumental tracks. This demonstrates a qualitative leap in generalization, enabling robust accompaniment for instruments - a task where existing models completely fail - and paving the way for more versatile music co-creation tools. Demo audio and code: https://anyaccomp.github.io