# Hybrid Transformers for Music Source Separation

arXiv: https://arxiv.org/abs/2211.08553 · HTML: https://mlsanity.com/paper/2211.08553/

## Key facts

- Cited by 134 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2022.

## Built on by (most cited first, 30 of 134)

- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [Moisesdb: A dataset for source separation beyond 4-stems](https://mlsanity.com/paper/2307.15913/) (2023)
- [MARBLE: Music Audio Representation Benchmark for Universal Evaluation](https://mlsanity.com/paper/2306.10548/) (2023)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation](https://mlsanity.com/paper/2502.13128/) (2025)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [SingFake: Singing Voice Deepfake Detection](https://mlsanity.com/paper/2309.07525/) (2023)
- [SONICS: Synthetic Or Not -- Identifying Counterfeit Songs](https://mlsanity.com/paper/2408.14080/) (2024)
- [X-Streamer: Unified Human World Modeling with Audiovisual Interaction](https://mlsanity.com/paper/2509.21574/) (2025)
- [JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation](https://mlsanity.com/paper/2310.19180/) (2023)
- [MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music](https://mlsanity.com/paper/2402.09871/) (2024)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos](https://mlsanity.com/paper/2409.07450/) (2024)
- [Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation](https://mlsanity.com/paper/2503.19611/) (2025)
- [MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions](https://mlsanity.com/paper/2407.20962/) (2024)
- [A Diffusion-Based Generative Equalizer for Music Restoration](https://mlsanity.com/paper/2403.18636/) (2024)
- [SongCreator: Lyrics-based Universal Song Generation](https://mlsanity.com/paper/2409.06029/) (2024)
- [MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation](https://mlsanity.com/paper/2407.15060/) (2024)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation](https://mlsanity.com/paper/2309.02539/) (2023)
- [OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup](https://mlsanity.com/paper/2410.21269/) (2024)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [AutoMV: An Automatic Multi-Agent System for Music Video Generation](https://mlsanity.com/paper/2512.12196/) (2025)
- [A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stems](https://mlsanity.com/paper/2406.18747/) (2024)
- [Benchmarks and leaderboards for sound demixing tasks](https://mlsanity.com/paper/2305.07489/) (2023)
- [GASS: Generalizing Audio Source Separation with Large-scale Data](https://mlsanity.com/paper/2310.00140/) (2023)

## Abstract (arXiv)

A natural question arising in Music Source Separation (MSS) is whether long range contextual information is useful, or whether local acoustic features are sufficient. In other fields, attention based Transformers have shown their ability to integrate information over long sequences. In this work, we introduce Hybrid Transformer Demucs (HT Demucs), an hybrid temporal/spectral bi-U-Net based on Hybrid Demucs, where the innermost layers are replaced by a cross-domain Transformer Encoder, using self-attention within one domain, and cross-attention across domains. While it performs poorly when trained only on MUSDB, we show that it outperforms Hybrid Demucs (trained on the same data) by 0.45 dB of SDR when using 800 extra training songs. Using sparse attention kernels to extend its receptive field, and per source fine-tuning, we achieve state-of-the-art results on MUSDB with extra training data, with 9.20 dB of SDR.