# Target Sound Extraction with Variable Cross-modality Clues

arXiv: https://arxiv.org/abs/2303.08372 · HTML: https://mlsanity.com/paper/2303.08372/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2023.

## Built on by (most cited first, 6 of 6)

- [SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer](https://mlsanity.com/paper/2409.08425/) (2024)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Language-based Audio Moment Retrieval](https://mlsanity.com/paper/2409.15672/) (2024)
- [Multichannel-to-Multichannel Target Sound Extraction Using Direction and Timestamp Clues](https://mlsanity.com/paper/2409.12415/) (2024)
- [SoundCompass: Navigating Target Sound Extraction With Effective Directional Clue Integration In Complex Acoustic Scenes](https://mlsanity.com/paper/2509.18561/) (2025)
- [Automatic Contextual Audio Denoising](https://mlsanity.com/paper/2605.22262/) (2026)

## Abstract (arXiv)

Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources. It often uses a model conditioned on a fixed form of target sound clues, such as a sound class label, which limits the ways in which users can interact with the model to specify the target sounds. To leverage variable number of clues cross modalities available in the inference phase, including a video, a sound event class, and a text caption, we propose a unified transformer-based TSE model architecture, where a multi-clue attention module integrates all the clues across the modalities. Since there is no off-the-shelf benchmark to evaluate our proposed approach, we build a dataset based on public corpora, Audioset and AudioCaps. Experimental results for seen and unseen target-sound evaluation sets show that our proposed TSE model can effectively deal with a varying number of clues which improves the TSE performance and robustness against partially compromised clues.