mlsanityData to 7 Oct 2026

Separate Anything You Describe

arXiv 2308.05037 ↗First posted 9 Aug 2023eess.AS cs.AI cs.MMcited by 43

Not on the map. Why?

Abstract (arXiv)

Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given a natural language query, which provides a natural and scalable interface for digital audio applications. Recent works on LASS, despite attaining promising separation performance on specific sources (e.g., musical instruments, limited classes of audio events), are unable to separate audio concepts in the open domain. In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries. We train AudioSep on large-scale multimodal datasets and extensively evaluate its capabilities on numerous tasks including audio event separation, musical instrument separation, and speech enhancement. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability using audio captions or text labels as queries, substantially outperforming previous audio-queried and language-queried sound separation models. For reproducibility of this work, we will release the source code, evaluation benchmark and pre-trained model at: https://github.com/Audio-AGI/AudioSep.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 73 of its 101 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

73 of 101 references.

References · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  8. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  11. 2018SDR - half-baked or well done?cited by 420
  12. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
38 more
  1. 2019Clotho: An Audio Captioning Datasetcited by 354
  2. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  3. 2022AudioGen: Textually Guided Audio Generationcited by 292
  4. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  5. 2018The Sound of Pixelscited by 258
  6. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  7. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  8. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  9. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  10. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  11. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  12. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  13. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  14. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  15. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  16. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  17. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  18. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  19. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  20. 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 53
  21. 2021Audio Captioning Transformercited by 53
  22. 2023MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectracited by 51
  23. 2021Audio Retrieval with Natural Language Queriescited by 49
  24. 2021Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separationcited by 48
  25. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  26. 2022CMGAN: Conformer-based Metric GAN for Speech Enhancementcited by 45
  27. 2019Universal Sound Separationcited by 41
  28. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  29. 2019Time Domain Audio Visual Speech Separationcited by 30
  30. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
  31. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  32. 2020Listen to What You Want: Neural Network-based Universal Sound Selectorcited by 27
  33. 2021Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Datacited by 27
  34. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  35. 2023Prefix tuning for automated audio captioningcited by 26
  36. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  37. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  38. 2018Improved Speech Enhancement with the Wave-U-Netcited by 23

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 16; 2026: 7.

43 papers.

Sort:

By area:

  1. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  2. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  3. 2024Taming Data and Transformers for Audio Generationcited by 28
  4. 2024FlowSep: Language-Queried Sound Separation with Rectified Flow Matchingcited by 21
  5. 2024Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labelingcited by 20
  6. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  7. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 15
  8. 2024SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformercited by 11
  9. 2025Guiding Audio Editing with Audio Language Modelcited by 11
  10. 2024A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stemscited by 7
  11. 2023GASS: Generalizing Audio Source Separation with Large-scale Datacited by 6
  12. 2023Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extractioncited by 6
31 more
  1. 2024Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experiencecited by 6
  2. 2024Learning Source Disentanglement in Neural Audio Codeccited by 6
  3. 2024Weakly-supervised Audio Separation via Bi-modal Semantic Similaritycited by 5
  4. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 5
  5. 2024Target conversation extraction: Source separation using turn-taking dynamicscited by 4
  6. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  7. 2025ZeroSep: Separate Anything in Audio with Zero Trainingcited by 4
  8. 2025FlexSED: Towards Open-Vocabulary Sound Event Detectioncited by 4
  9. 2025Event2Audio: Event-Based Optical Vibration Sensingcited by 3
  10. 2025MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separationcited by 3
  11. 2024Multichannel-to-Multichannel Target Sound Extraction Using Direction and Timestamp Cluescited by 2
  12. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  13. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimizationcited by 2
  14. 2025Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidancecited by 2
  15. 2025Neural Speech Extraction with Human Feedbackcited by 2
  16. 2024Universal Sound Separation with Self-Supervised Audio Masked Autoencodercited by 1
  17. 2024Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairscited by 1
  18. 2024EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generationcited by 1
  19. 2025Unleashing the Power of Natural Audio Featuring Multiple Sound Sourcescited by 1
  20. 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separationcited by 1
  21. 2026SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separationcited by 1
  22. 2026SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarioscited by 1
  23. 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilitiescited by 1
  24. 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
  25. 2024Performance Improvement of Language-Queried Audio Source Separation Based on Caption Augmentation From Large Language Models for DCASE Challenge 2024 Task 9cited by 0
  26. 2024Did You Hear That? Introducing AADG: A Framework for Generating Benchmark Data in Audio Anomaly Detectioncited by 0
  27. 2025Text-Queried Audio Source Separation via Hierarchical Modelingcited by 0
  28. 2025Generative Multi-modal Feedback for Singing Voice Synthesis Evaluationcited by 0
  29. 2026SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Trainingcited by 0
  30. 2026VorTEX: Various overlap ratio for Target speech EXtractioncited by 0
  31. 2026Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Modelscited by 0