mlsanityData to 7 Oct 2026

Audio Mamba: Bidirectional State Space Model for Audio Representation Learning

arXiv 2406.03344 ↗First posted 5 Jun 2024cs.SD cs.AI eess.AScited by 36

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

Transformers have rapidly become the preferred choice for audio classification, surpassing methods based on CNNs. However, Audio Spectrogram Transformers (ASTs) exhibit quadratic scaling due to self-attention. The removal of this quadratic self-attention cost presents an appealing direction. Recently, state space models (SSMs), such as Mamba, have demonstrated potential in language and vision tasks in this regard. In this study, we explore whether reliance on self-attention is necessary for audio classification tasks. By introducing Audio Mamba (AuM), the first self-attention-free, purely SSM-based model for audio classification, we aim to address this question. We evaluate AuM on various audio datasets - comprising six different benchmarks - where it achieves comparable or better performance compared to well-established AST model.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 34 of its 36 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

34 of 36 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  8. 2024VMamba: Visual State Space Modelcited by 955
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  10. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
22 more
  1. 2021AST: Audio Spectrogram Transformercited by 634
  2. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  3. 2020Rescaling Egocentric Visioncited by 529
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  6. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  7. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  8. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  9. 2022Masked Autoencoders that Listencited by 271
  10. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  11. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  12. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  13. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  14. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  15. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  16. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  17. 2024Vivim: a Video Vision Mamba for Medical Video Segmentationcited by 51
  18. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  19. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  20. 2022UAVM: Towards Unifying Audio and Visual Modelscited by 20
  21. 2023A Neural State-Space Model Approach to Efficient Speech Separationcited by 6
  22. 2023Transcription free filler word detection with Neural semi-CRFscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 18; 2026: 3.

36 papers.

Sort:

By area:

  1. 2024Foundation Models for Music: A Surveycited by 34
  2. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  3. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  4. 2024DepMamba: Progressive Fusion Mamba for Multimodal Depression Detectioncited by 14
  5. 2025Latent Mamba Operator for Partial Differential Equationscited by 8
  6. 2025Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternativecited by 6
  7. 2024TF-Mamba: A Time-Frequency Network for Sound Source Localizationcited by 5
  8. 2024DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classificationcited by 3
  9. 2024Mamba-based Segmentation Model for Speaker Diarizationcited by 3
  10. 2024Toward Physics-guided Time Series Embeddingcited by 3
  11. 2024State-Space Large Audio Language Modelscited by 3
  12. 2025Efficient Spiking Point Mamba for Point Cloud Analysiscited by 3
24 more
  1. 2025Exploring Audio Cues for Enhanced Test-Time Video Model Adaptationcited by 3
  2. 2025SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentationcited by 3
  3. 2024Self-supervised Learning for Acoustic Few-Shot Classificationcited by 2
  4. 2024TAME: Temporal Audio-based Mamba for Enhanced Drone Trajectory Estimation and Classificationcited by 2
  5. 2025Deep Active Speech Cancellation with Mamba-Masking Networkcited by 2
  6. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  7. 2024Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layerscited by 1
  8. 2024BadScan: An Architectural Backdoor Attack on Visual State Space Modelscited by 1
  9. 2024Exploring Applications of State Space Models and Advanced Training Techniques in Sequential Recommendations: A Comparative Study on Efficiency and Performancecited by 0
  10. 2024SambaMixer: State of Health Prediction of Li-ion Batteries using Mamba State Space Modelscited by 0
  11. 2025GLADMamba: Unsupervised Graph-Level Anomaly Detection Powered by Selective State Space Modelcited by 0
  12. 2025Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberationcited by 0
  13. 2025PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognitioncited by 0
  14. 2025Recognizing Dementia from Neuropsychological Tests with State Space Modelscited by 0
  15. 2025Keyword Mamba: Spoken Keyword Spotting with State Space Modelscited by 0
  16. 2025Towards Efficient Vision State Space Models via Token Mergingcited by 0
  17. 2025Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classificationcited by 0
  18. 2025ESTM: An Enhanced Dual-Branch Spectral-Temporal Mamba for Anomalous Sound Detectioncited by 0
  19. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  20. 2025Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mambacited by 0
  21. 2025PSMamba: Progressive Self-supervised Vision Mamba for Plant Disease Recognitioncited by 0
  22. 2026SqueezeComposer: Temporal Speed-up is A Simple Trick for Long-form Music Composingcited by 0
  23. 2026HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limitcited by 0
  24. 2026AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
  2. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model
  3. 2024Audio Mamba: Pretrained Audio State Space Model For Audio Tagging
  4. 2025SAM: A Mamba-2 State-Space Audio-Language Model
  5. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
  6. 2025ASCMamba: Multimodal Time-Frequency Mamba for Acoustic Scene Classification
  7. 2026Omni-directional attention mechanism based on Mamba for speech separation
  8. 2024Mamba in Speech: Towards an Alternative to Self-Attention