# Cross-attention Inspired Selective State Space Models for Target Sound Extraction

arXiv: https://arxiv.org/abs/2409.04803 · HTML: https://mlsanity.com/paper/2409.04803/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 7 Sep 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline](https://mlsanity.com/paper/1807.09902/) (2018)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakers](https://mlsanity.com/paper/2403.07675/) (2024)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Exploring the Capability of Mamba in Speech Applications](https://mlsanity.com/paper/2406.16808/) (2024)
- [One-shot conditional audio filtering of arbitrary sounds](https://mlsanity.com/paper/2011.02421/) (2020)
- [AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction](https://mlsanity.com/paper/2306.14170/) (2023)
- [SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning](https://mlsanity.com/paper/2204.03895/) (2022)
- [Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2404.12725/) (2024)
- [TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion](https://mlsanity.com/paper/2401.14185/) (2024)
- [TSE-PI: Target Sound Extraction under Reverberant Environments with Pitch Information](https://mlsanity.com/paper/2406.08716/) (2024)

## Built on by (most cited first, 4 of 4)

- [Multi-Agent Reinforcement Learning with Selective State-Space Models](https://mlsanity.com/paper/2410.19382/) (2024)
- [MambaFlow: A Mamba-Centric Architecture for End-to-End Optical Flow Estimation](https://mlsanity.com/paper/2503.07046/) (2025)
- [Cross-attention for State-based model RWKV-7](https://mlsanity.com/paper/2504.14260/) (2025)
- [MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal Prostate MRI Segmentation](https://mlsanity.com/paper/2510.17529/) (2025)

## Abstract (arXiv)

The Transformer model, particularly its cross-attention module, is widely used for feature fusion in target sound extraction which extracts the signal of interest based on given clues. Despite its effectiveness, this approach suffers from low computational efficiency. Recent advancements in state space models, notably the latest work Mamba, have shown comparable performance to Transformer-based methods while significantly reducing computational complexity in various tasks. However, Mamba's applicability in target sound extraction is limited due to its inability to capture dependencies between different sequences as the cross-attention does. In this paper, we propose CrossMamba for target sound extraction, which leverages the hidden attention mechanism of Mamba to compute dependencies between the given clues and the audio mixture. The calculation of Mamba can be divided to the query, key and value. We utilize the clue to generate the query and the audio mixture to derive the key and value, adhering to the principle of the cross-attention mechanism in Transformers. Experimental results from two representative target sound extraction methods validate the efficacy of the proposed CrossMamba.