# SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model

arXiv: https://arxiv.org/abs/2405.11831 · HTML: https://mlsanity.com/paper/2405.11831/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 20 May 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining](https://mlsanity.com/paper/2402.03302/) (2024)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2402.00789/) (2024)
- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakers](https://mlsanity.com/paper/2403.07675/) (2024)
- [TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms](https://mlsanity.com/paper/2405.01242/) (2024)

## Built on by (most cited first, 30 of 37)

- [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis](https://mlsanity.com/paper/2407.09732/) (2024)
- [Long-Form Speech Generation with Spoken Language Models](https://mlsanity.com/paper/2412.18603/) (2024)
- [Micro-gesture Online Recognition using Learnable Query Points](https://mlsanity.com/paper/2407.04490/) (2024)
- [Rethinking Mamba in Speech Processing by Self-Supervised Models](https://mlsanity.com/paper/2409.07273/) (2024)
- [TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification](https://mlsanity.com/paper/2406.04419/) (2024)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [TF-Mamba: A Time-Frequency Network for Sound Source Localization](https://mlsanity.com/paper/2409.05034/) (2024)
- [MECG-E: Mamba-based ECG Enhancer for Baseline Wander Removal](https://mlsanity.com/paper/2409.18828/) (2024)
- [Exploring Finetuned Audio-LLM on Heart Murmur Features](https://mlsanity.com/paper/2501.13884/) (2025)
- [Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching](https://mlsanity.com/paper/2410.06285/) (2024)
- [Mamba-based Segmentation Model for Speaker Diarization](https://mlsanity.com/paper/2410.06459/) (2024)
- [State-Space Large Audio Language Models](https://mlsanity.com/paper/2411.15685/) (2024)
- [Self-supervised Learning for Acoustic Few-Shot Classification](https://mlsanity.com/paper/2409.09647/) (2024)
- [Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity](https://mlsanity.com/paper/2410.06846/) (2024)
- [MSEMG: Surface Electromyography Denoising with a Mamba-based Efficient Network](https://mlsanity.com/paper/2411.18902/) (2024)
- [TAME: Temporal Audio-based Mamba for Enhanced Drone Trajectory Estimation and Classification](https://mlsanity.com/paper/2412.13037/) (2024)
- [Deep Active Speech Cancellation with Mamba-Masking Network](https://mlsanity.com/paper/2502.01185/) (2025)
- [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025)
- [Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models](https://mlsanity.com/paper/2602.20981/) (2026)
- [Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba](https://mlsanity.com/paper/2406.16722/) (2024)
- [Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models](https://mlsanity.com/paper/2412.08595/) (2024)
- [Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberation](https://mlsanity.com/paper/2505.16911/) (2025)
- [PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition](https://mlsanity.com/paper/2506.01138/) (2025)
- [Recognizing Dementia from Neuropsychological Tests with State Space Models](https://mlsanity.com/paper/2507.10311/) (2025)
- [ESTM: An Enhanced Dual-Branch Spectral-Temporal Mamba for Anomalous Sound Detection](https://mlsanity.com/paper/2509.02471/) (2025)
- [An overview of neural architectures for self-supervised audio representation learning from masked spectrograms](https://mlsanity.com/paper/2509.18691/) (2025)
- [Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba](https://mlsanity.com/paper/2510.04738/) (2025)
- [Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank](https://mlsanity.com/paper/2510.10948/) (2025)

## Abstract (arXiv)

Transformers have revolutionized deep learning across various tasks, including audio representation learning, due to their powerful modeling capabilities. However, they often suffer from quadratic complexity in both GPU memory usage and computational inference time, affecting their efficiency. Recently, state space models (SSMs) like Mamba have emerged as a promising alternative, offering a more efficient approach by avoiding these complexities. Given these advantages, we explore the potential of SSM-based models in audio tasks. In this paper, we introduce Self-Supervised Audio Mamba (SSAMBA), the first self-supervised, attention-free, and SSM-based model for audio representation learning. SSAMBA leverages the bidirectional Mamba to capture complex audio patterns effectively. We incorporate a self-supervised pretraining framework that optimizes both discriminative and generative objectives, enabling the model to learn robust audio representations from large-scale, unlabeled datasets. We evaluated SSAMBA on various tasks such as audio classification, keyword spotting, and speaker identification. Our results demonstrate that SSAMBA outperforms the Self-Supervised Audio Spectrogram Transformer (SSAST) in most tasks. Notably, SSAMBA is approximately 92.7% faster in batch inference speed and 95.4% more memory-efficient than SSAST for the tiny model size with an input token size of 22k. These efficiency gains, combined with superior performance, underscore the effectiveness of SSAMBA's architectural innovation, making it a compelling choice for a wide range of audio processing applications.