mlsanityData to 14 Oct 2026

SAM: A Mamba-2 State-Space Audio-Language Model

arXiv 2509.15680 ↗First posted 19 Sep 2025cs.SD eess.AScited by 0

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

We present SAM, a State-space Audio-language Model that integrates an audio encoder with a Mamba-2 backbone. SAM-2.7B achieves 21.1 mAP on AudioSet and 17.6 SPICE on AudioCaps, matching or surpassing larger 7B transformer-based models with fewer parameters. We further provide the first systematic, representation-level analysis of how SSMs interact with audio encoder outputs: (1) joint audio encoder finetuning is essential, supported by accuracy gains and observed adaptation of token representation rank and similarity across different SSM sizes; (2) despite linear scaling, SSMs benefit more from compact, information-rich audio token representations than from excessively long token sequences; and (3) incorporating instruction-following supervision substantially improves reasoning ability, boosting MMAU-Sound accuracy from 22.8 to 56.8. Through comprehensive experiments and analysis, we establish practical design principles for SSMs as strong, scalable backbones for audio-language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 28 of its 42 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

28 of 42 references.

References · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  7. 2024VMamba: Visual State Space Modelcited by 969
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  10. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  11. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  12. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
16 more
  1. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
  2. 2023Listen, Think, and Understandcited by 209
  3. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  4. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  5. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  6. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  7. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  8. 2019Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimizationcited by 29
  9. 2025Mellow: a small audio language model for reasoningcited by 27
  10. 2024SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMscited by 15
  11. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  12. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  13. 2023Representation Deficiency in Masked Language Modelingcited by 10
  14. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modelingcited by 6
  15. 2024FaceGPT: Self-supervised Learning to Chat about 3D Human Facescited by 4
  16. 2024State-Space Large Audio Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
  2. 2025U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding
  3. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model
  4. 2026Eureka-Audio: Triggering Audio Intelligence in Compact Language Models
  5. 2026Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
  6. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
  7. 2025Exploring State-Space-Model based Language Model in Music Generation
  8. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling