mlsanityData to 7 Oct 2026

Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations

arXiv 2406.02178 ↗First posted 4 Jun 2024cs.SD cs.AI eess.AScited by 19

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

Despite its widespread adoption as the prominent neural architecture, the Transformer has spurred several independent lines of work to address its limitations. One such approach is selective state space models, which have demonstrated promising results for language modelling. However, their feasibility for learning self-supervised, general-purpose audio representations is yet to be investigated. This work proposes Audio Mamba, a selective state space model for learning general-purpose audio representations from randomly masked spectrogram patches through self-supervision. Empirical results on ten diverse audio recognition downstream tasks show that the proposed models, pretrained on the AudioSet dataset, consistently outperform comparable self-supervised audio spectrogram transformer (SSAST) baselines by a considerable margin and demonstrate better performance in dataset size, sequence length and model size comparisons.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 27 of its 31 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

27 of 31 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
15 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2022Masked Autoencoders that Listencited by 271
  9. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  10. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  11. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  12. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  13. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  14. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  15. 2023HyperConformer: Multi-head HyperMixer for Efficient Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  2. 2025Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternativecited by 6
  3. 2024Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matchingcited by 4
  4. 2025xLSTM-SENet: xLSTM for Single-Channel Speech Enhancementcited by 4
  5. 2024MambaFoley: Foley Sound Generation using Selective State-Space Modelscited by 3
  6. 2024Mamba-based Segmentation Model for Speaker Diarizationcited by 3
  7. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Modelscited by 3
  8. 2025Deep Active Speech Cancellation with Mamba-Masking Networkcited by 2
  9. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  10. 2025MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancementcited by 2
  11. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 1
  12. 2026Elastic Spectral State Space Models for Train-Once Budgeted Inferencecited by 1
7 more
  1. 2025Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberationcited by 0
  2. 2025GRAM: Spatial general-purpose audio representation models for real-world applicationscited by 0
  3. 2025PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognitioncited by 0
  4. 2025Are Mamba-based Audio Foundation Models the Best Fit for Non-Verbal Emotion Recognition?cited by 0
  5. 2025AudioMAE++: learning better masked audio representations with SwiGLU FFNscited by 0
  6. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  7. 2026MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audiocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learning
  2. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model
  3. 2024Audio Mamba: Pretrained Audio State Space Model For Audio Tagging
  4. 2025SAM: A Mamba-2 State-Space Audio-Language Model
  5. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
  6. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
  7. 2024Rethinking Mamba in Speech Processing by Self-Supervised Models
  8. 2024Self-supervised Learning for Acoustic Few-Shot Classification