# Text-Driven Separation of Arbitrary Sounds

arXiv: https://arxiv.org/abs/2204.05738 · HTML: https://mlsanity.com/paper/2204.05738/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2022.

## Built on by (most cited first, 23 of 23)

- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022)
- [CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos](https://mlsanity.com/paper/2212.07065/) (2022)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Language-Guided Audio-Visual Source Separation via Trimodal Consistency](https://mlsanity.com/paper/2303.16342/) (2023)
- [Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables](https://mlsanity.com/paper/2311.00320/) (2023)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience](https://mlsanity.com/paper/2402.03710/) (2024)
- [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022)
- [Weakly-supervised Audio Separation via Bi-modal Semantic Similarity](https://mlsanity.com/paper/2404.01740/) (2024)
- [MAQA: A Multimodal QA Benchmark for Negation](https://mlsanity.com/paper/2301.03238/) (2023)
- [MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction](https://mlsanity.com/paper/2505.23305/) (2025)
- [OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation](https://mlsanity.com/paper/2409.19270/) (2024)
- [SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction](https://mlsanity.com/paper/2506.00273/) (2025)
- [Recomposer: Event-roll-guided generative audio editing](https://mlsanity.com/paper/2509.05256/) (2025)
- [FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation](https://mlsanity.com/paper/2608.22111/) (2026)
- [Complete and separate: Conditional separation with missing target source attribute completion](https://mlsanity.com/paper/2307.14609/) (2023)
- [CATSE: A Context-Aware Framework for Causal Target Sound Extraction](https://mlsanity.com/paper/2403.14246/) (2024)
- [Text-Queried Audio Source Separation via Hierarchical Modeling](https://mlsanity.com/paper/2505.21025/) (2025)
- [Fine-grained Soundscape Control for Augmented Hearing](https://mlsanity.com/paper/2603.00395/) (2026)
- [Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models](https://mlsanity.com/paper/2609.36577/) (2026)

## Abstract (arXiv)

We propose a method of separating a desired sound source from a single-channel mixture, based on either a textual description or a short audio sample of the target source. This is achieved by combining two distinct models. The first model, SoundWords, is trained to jointly embed both an audio clip and its textual description to the same embedding in a shared representation. The second model, SoundFilter, takes a mixed source audio clip as an input and separates it based on a conditioning vector from the shared text-audio representation defined by SoundWords, making the model agnostic to the conditioning modality. Evaluating on multiple datasets, we show that our approach can achieve an SI-SDR of 9.1 dB for mixtures of two arbitrary sounds when conditioned on text and 10.1 dB when conditioned on audio. We also show that SoundWords is effective at learning co-embeddings and that our multi-modal training approach improves the performance of SoundFilter.