# Multichannel-to-Multichannel Target Sound Extraction Using Direction and Timestamp Clues

arXiv: https://arxiv.org/abs/2409.12415 · HTML: https://mlsanity.com/paper/2409.12415/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline](https://mlsanity.com/paper/1807.09902/) (2018)
- [Listen to What You Want: Neural Network-based Universal Sound Selector](https://mlsanity.com/paper/2006.05712/) (2020)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Dense CNN with Self-Attention for Time-Domain Speech Enhancement](https://mlsanity.com/paper/2009.01941/) (2020)
- [SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning](https://mlsanity.com/paper/2204.03895/) (2022)
- [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022)
- [Enhancing End-to-End Multi-channel Speech Separation via Spatial Feature Learning](https://mlsanity.com/paper/2003.03927/) (2020)
- [End-to-End Multi-Channel Speech Separation](https://mlsanity.com/paper/1905.06286/) (2019)
- [Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables](https://mlsanity.com/paper/2311.00320/) (2023)
- [Improving Target Sound Extraction with Timestamp Information](https://mlsanity.com/paper/2204.00821/) (2022)
- [Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters](https://mlsanity.com/paper/2304.12023/) (2023)
- [ReZero: Region-customizable Sound Extraction](https://mlsanity.com/paper/2308.16892/) (2023)
- [CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction](https://mlsanity.com/paper/2402.17455/) (2024)
- [Multi-Microphone Complex Spectral Mapping for Speech Dereverberation](https://mlsanity.com/paper/2003.01861/) (2020)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments](https://mlsanity.com/paper/2011.05958/) (2020)
- [All Neural Low-latency Directional Speech Extraction](https://mlsanity.com/paper/2407.04879/) (2024)
- [Target Sound Extraction with Variable Cross-modality Clues](https://mlsanity.com/paper/2303.08372/) (2023)
- [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022)
- [DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement](https://mlsanity.com/paper/2212.07570/) (2022)
- [DeFTAN-II: Efficient Multichannel Speech Enhancement with Subgroup Processing](https://mlsanity.com/paper/2308.15777/) (2023)
- [Binaural Speech Enhancement Using Deep Complex Convolutional Transformer Networks](https://mlsanity.com/paper/2403.05393/) (2024)

## Built on by (most cited first, 2 of 2)

- [Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers](https://mlsanity.com/paper/2603.23723/) (2026)
- [SoundCompass: Navigating Target Sound Extraction With Effective Directional Clue Integration In Complex Acoustic Scenes](https://mlsanity.com/paper/2509.18561/) (2025)

## Abstract (arXiv)

We propose a multichannel-to-multichannel target sound extraction (M2M-TSE) framework for separating multichannel target signals from a multichannel mixture of sound sources. Target sound extraction (TSE) isolates a specific target signal using user-provided clues, typically focusing on single-channel extraction with class labels or temporal activation maps. However, to preserve and utilize spatial information in multichannel audio signals, it is essential to extract multichannel signals of a target sound source. Moreover, the clue for extraction can also include spatial or temporal cues like direction-of-arrival (DoA) or timestamps of source activation. To address these challenges, we present an M2M framework that extracts a multichannel sound signal based on spatio-temporal clues. We demonstrate that our transformer-based architecture can successively accomplish the M2M-TSE task for multichannel signals synthesized from audio signals of diverse classes in different room environments. Furthermore, we show that the multichannel extraction task introduces sufficient inductive bias in the DNN, allowing it to directly handle DoA clues without utilizing hand-crafted spatial features.