mlsanityData to 7 Oct 2026

Real-Time Target Sound Extraction

arXiv 2211.02250 ↗First posted 4 Nov 2022cs.SD cs.LG eess.AScited by 22

Not on the map. Why?

Abstract (arXiv)

We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner while also leveraging the generalization performance of transformer-based architectures. Our evaluations show as much as 2.2-3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2-4x smaller model size and a 1.5-2x lower runtime. We provide code, dataset, and audio samples: https://waveformer.cs.washington.edu/.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

36 of 42 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2018SDR - half-baked or well done?cited by 420
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  6. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  7. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  8. 2020Attention is All You Need in Speech Separationcited by 184
  9. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  10. 2019Co-Separating Sounds of Visual Objectscited by 110
  11. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  12. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
24 more
  1. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 72
  2. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  3. 2016Single-Channel Multi-Speaker Separation using Deep Clusteringcited by 45
  4. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  5. 2019Universal Sound Separationcited by 41
  6. 2018Differentiable Consistency Constraints for Improved Deep Speech Enhancementcited by 35
  7. 2018General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baselinecited by 30
  8. 2020Listen to What You Want: Neural Network-based Universal Sound Selectorcited by 27
  9. 2021Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Datacited by 27
  10. 2022Text-Driven Separation of Arbitrary Soundscited by 23
  11. 2020One-shot conditional audio filtering of arbitrary soundscited by 18
  12. 2021Personalized Speech Enhancement: New Models and Comprehensive Evaluationcited by 18
  13. 2022SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learningcited by 14
  14. 2022Heterogeneous Target Speech Separationcited by 13
  15. 2020Source separation with weakly labelled data: An approach to computational auditory scene analysiscited by 12
  16. 2022Resource-Efficient Separation Transformercited by 11
  17. 2021Personalized PercepNet: Real-time, Low-complexity Target Voice Separation and Enhancementcited by 9
  18. 2021Hybrid Neural Networks for On-device Directional Hearingcited by 8
  19. 2022Tiny-Sepformer: A Tiny Time-Domain Transformer Network for Speech Separationcited by 8
  20. 2019WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separationcited by 7
  21. 2022ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancementcited by 7
  22. 2021Few-shot learning of new sound classes for target sound extractioncited by 5
  23. 2021Environmental Sound Extraction Using Onomatopoeic Wordscited by 3
  24. 2022Semi-supervised Time Domain Target Speaker Extraction with Attentioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 8; 2026: 2.

22 papers.

Sort:

By area:

  1. 2025SuperBPE: Space Travel for Language Modelscited by 58
  2. 2023Separate Anything You Describecited by 43
  3. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extractioncited by 12
  4. 2024Look Once to Hear: Target Speech Hearing with Noisy Examplescited by 11
  5. 2024SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformercited by 11
  6. 2024OmniSep: Unified Omni-Modality Sound Separation with Query-Mixupcited by 8
  7. 2023Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extractioncited by 6
  8. 2024Knowledge boosting during low-latency inferencecited by 4
  9. 2024Target conversation extraction: Source separation using turn-taking dynamicscited by 4
  10. 2024Cross-attention Inspired Selective State Space Models for Target Sound Extractioncited by 4
  11. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  12. 2024DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classificationcited by 3
10 more
  1. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  2. 2024Multichannel-to-Multichannel Target Sound Extraction Using Direction and Timestamp Cluescited by 2
  3. 2025SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extractioncited by 2
  4. 2025Neural Speech Extraction with Human Feedbackcited by 2
  5. 2025TF-MLPNet: Tiny Real-Time Neural Speech Separationcited by 1
  6. 2026MoXaRt: Audio-Visual Object-Guided Sound Interaction for XRcited by 1
  7. 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
  8. 2024CATSE: A Context-Aware Framework for Causal Target Sound Extractioncited by 0
  9. 2025Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separationcited by 0
  10. 2025Text-Queried Audio Source Separation via Hierarchical Modelingcited by 0