mlsanityData to 7 Oct 2026

SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer

arXiv 2409.08425 ↗First posted 12 Sep 2024eess.AS cs.SDcited by 11

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this paper, we introduce SoloAudio, a novel diffusion-based generative model for target sound extraction (TSE). Our approach trains latent diffusion models on audio, replacing the previous U-Net backbone with a skip-connected Transformer that operates on latent features. SoloAudio supports both audio-oriented and language-oriented TSE by utilizing a CLAP model as the feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic audio generated by state-of-the-art text-to-audio models for training, demonstrating strong generalization to out-of-domain data and unseen sound events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and real data from AudioSet, where SoloAudio achieves the state-of-the-art results on both in-domain and out-of-domain data, and exhibits impressive zero-shot and few-shot capabilities. Source code and demos are released.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 28 of its 33 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

28 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Noise-robust Speech Separation with Fast Generative Correction11 Jun
  2. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extraction6 Oct
  3. 2023Target Speech Extraction with Conditional Diffusion Model8 Aug
  4. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separation4 Feb
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 23

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  6. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  7. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  9. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  10. 2022gDDIM: Generalized denoising diffusion implicit modelscited by 125
  11. 2019Co-Separating Sounds of Visual Objectscited by 110
  12. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
11 more
  1. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  2. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  3. 2023Separate Anything You Describecited by 43
  4. 2020Listen to What You Want: Neural Network-based Universal Sound Selectorcited by 27
  5. 2022Real-Time Target Sound Extractioncited by 22
  6. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19
  7. 2020One-shot conditional audio filtering of arbitrary soundscited by 18
  8. 2022SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learningcited by 14
  9. 2022Improving Target Sound Extraction with Timestamp Informationcited by 10
  10. 2023Target Sound Extraction with Variable Cross-modality Cluescited by 6
  11. 2021Few-shot learning of new sound classes for target sound extractioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipelinecited by 9
  2. 2025FlexSED: Towards Open-Vocabulary Sound Event Detectioncited by 4
  3. 2025Event2Audio: Event-Based Optical Vibration Sensingcited by 3
  4. 2025AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusioncited by 2
  5. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimizationcited by 2
  6. 2025SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filteringcited by 1
  7. 2026SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separationcited by 1
  8. 2026MoXaRt: Audio-Visual Object-Guided Sound Interaction for XRcited by 1
  9. 2026StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Modelcited by 1
  10. 2026Fine-grained Soundscape Control for Augmented Hearingcited by 0
  11. 2026DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  2. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
  3. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data
  4. 2026SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation
  5. 2026FSD50K-Solo: Automated Curation of Single-Source Sound Events
  6. 2023ArchiSound: Audio Generation with Diffusion
  7. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
  8. 2023EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis