# CATSE: A Context-Aware Framework for Causal Target Sound Extraction

arXiv: https://arxiv.org/abs/2403.14246 · HTML: https://mlsanity.com/paper/2403.14246/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 17 of 17)

- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Asteroid: the PyTorch-based audio source separation toolkit for researchers](https://mlsanity.com/paper/2005.04132/) (2020)
- [TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids](https://mlsanity.com/paper/2005.11138/) (2020)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline](https://mlsanity.com/paper/1807.09902/) (2018)
- [Listen to What You Want: Neural Network-based Universal Sound Selector](https://mlsanity.com/paper/2006.05712/) (2020)
- [Text-Driven Separation of Arbitrary Sounds](https://mlsanity.com/paper/2204.05738/) (2022)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Conditioned Source Separation for Music Instrument Performances](https://mlsanity.com/paper/2004.03873/) (2020)
- [One-shot conditional audio filtering of arbitrary sounds](https://mlsanity.com/paper/2011.02421/) (2020)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning](https://mlsanity.com/paper/2204.03895/) (2022)
- [Source separation with weakly labelled data: An approach to computational auditory scene analysis](https://mlsanity.com/paper/2002.02065/) (2020)

## Abstract (arXiv)

Target Sound Extraction (TSE) focuses on the problem of separating sources of interest, indicated by a user's cue, from the input mixture. Most existing solutions operate in an offline fashion and are not suited to the low-latency causal processing constraints imposed by applications in live-streamed content such as augmented hearing. We introduce a family of context-aware low-latency causal TSE models suitable for real-time processing. First, we explore the utility of context by providing the TSE model with oracle information about what sound classes make up the input mixture, where the objective of the model is to extract one or more sources of interest indicated by the user. Since the practical applications of oracle models are limited due to their assumptions, we introduce a composite multi-task training objective involving separation and classification losses. Our evaluation involving single- and multi-source extraction shows the benefit of using context information in the model either by means of providing full context or via the proposed multi-task training loss without the need for full context information. Specifically, we show that our proposed model outperforms size- and latency-matched Waveformer, a state-of-the-art model for real-time TSE.