# Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing

arXiv: https://arxiv.org/abs/2609.29169 · HTML: https://mlsanity.com/paper/2609.29169/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [Synchformer: Efficient Synchronization from Sparse Cues](https://mlsanity.com/paper/2401.16423/) (2024)
- [Condensed Movies: Story Based Retrieval with Contextual Embeddings](https://mlsanity.com/paper/2005.04208/) (2020)
- [Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation](https://mlsanity.com/paper/2406.02347/) (2024)
- [Temporally Aligned Audio for Video with Autoregression](https://mlsanity.com/paper/2409.13689/) (2024)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [SAM Audio: Segment Anything in Audio](https://mlsanity.com/paper/2512.18099/) (2025)

## Abstract (arXiv)

We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: https://sse-ai.notion.site