# Audio Mamba: Bidirectional State Space Model for Audio Representation Learning

arXiv: https://arxiv.org/abs/2406.03344 · HTML: https://mlsanity.com/paper/2406.03344/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 34 of its 36 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [Vivim: a Video Vision Mamba for Medical Video Segmentation](https://mlsanity.com/paper/2401.14168/) (2024)
- [Epic-Sounds: A Large-scale Dataset of Actions That Sound](https://mlsanity.com/paper/2302.00646/) (2023)

## Built on by (most cited first, 30 of 37)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis](https://mlsanity.com/paper/2407.09732/) (2024)
- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection](https://mlsanity.com/paper/2409.15936/) (2024)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [TF-Mamba: A Time-Frequency Network for Sound Source Localization](https://mlsanity.com/paper/2409.05034/) (2024)
- [Mamba-based Segmentation Model for Speaker Diarization](https://mlsanity.com/paper/2410.06459/) (2024)
- [Efficient Spiking Point Mamba for Point Cloud Analysis](https://mlsanity.com/paper/2504.14371/) (2025)
- [DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification](https://mlsanity.com/paper/2409.12413/) (2024)
- [Toward Physics-guided Time Series Embedding](https://mlsanity.com/paper/2410.06651/) (2024)
- [State-Space Large Audio Language Models](https://mlsanity.com/paper/2411.15685/) (2024)
- [Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation](https://mlsanity.com/paper/2506.12481/) (2025)
- [SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation](https://mlsanity.com/paper/2510.10160/) (2025)
- [Self-supervised Learning for Acoustic Few-Shot Classification](https://mlsanity.com/paper/2409.09647/) (2024)
- [TAME: Temporal Audio-based Mamba for Enhanced Drone Trajectory Estimation and Classification](https://mlsanity.com/paper/2412.13037/) (2024)
- [Deep Active Speech Cancellation with Mamba-Masking Network](https://mlsanity.com/paper/2502.01185/) (2025)
- [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025)
- [Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers](https://mlsanity.com/paper/2411.08909/) (2024)
- [BadScan: An Architectural Backdoor Attack on Visual State Space Models](https://mlsanity.com/paper/2411.17283/) (2024)
- [Exploring Applications of State Space Models and Advanced Training Techniques in Sequential Recommendations: A Comparative Study on Efficiency and Performance](https://mlsanity.com/paper/2408.05606/) (2024)
- [SambaMixer: State of Health Prediction of Li-ion Batteries using Mamba State Space Models](https://mlsanity.com/paper/2411.00233/) (2024)
- [GLADMamba: Unsupervised Graph-Level Anomaly Detection Powered by Selective State Space Model](https://mlsanity.com/paper/2503.17903/) (2025)
- [Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberation](https://mlsanity.com/paper/2505.16911/) (2025)
- [PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition](https://mlsanity.com/paper/2506.01138/) (2025)
- [Recognizing Dementia from Neuropsychological Tests with State Space Models](https://mlsanity.com/paper/2507.10311/) (2025)
- [Keyword Mamba: Spoken Keyword Spotting with State Space Models](https://mlsanity.com/paper/2508.07363/) (2025)
- [Towards Efficient Vision State Space Models via Token Merging](https://mlsanity.com/paper/2508.13599/) (2025)
- [Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification](https://mlsanity.com/paper/2508.21243/) (2025)
- [ESTM: An Enhanced Dual-Branch Spectral-Temporal Mamba for Anomalous Sound Detection](https://mlsanity.com/paper/2509.02471/) (2025)

## Abstract (arXiv)

Transformers have rapidly become the preferred choice for audio classification, surpassing methods based on CNNs. However, Audio Spectrogram Transformers (ASTs) exhibit quadratic scaling due to self-attention. The removal of this quadratic self-attention cost presents an appealing direction. Recently, state space models (SSMs), such as Mamba, have demonstrated potential in language and vision tasks in this regard. In this study, we explore whether reliance on self-attention is necessary for audio classification tasks. By introducing Audio Mamba (AuM), the first self-attention-free, purely SSM-based model for audio classification, we aim to address this question. We evaluate AuM on various audio datasets - comprising six different benchmarks - where it achieves comparable or better performance compared to well-established AST model.