mlsanityData to 14 Oct 2026

DAVSS: Distilled Audio-Visual State Space Models

arXiv 2608.19523 ↗First posted 20 Aug 2026eess.AScited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

State-space models (SSMs) distilled from transformer teachers combine the performance of transformers with the efficiency of SSMs. We extend the Transformer-SSM knowledge distillation to a multimodal setting and propose the Distilled Audio-visual State-Space (DAVSS) model. The DAVSS model, 14M parameters, is 12 times smaller compared to transformer-based models such as CAV-MAE, and still outperforms them. DAVSS improves over the existing audio-visual models by: 1) Finer input resolution: using smaller patch sizes process the input, compensating for the smaller model size by increasing input sequence lengths. This is supported by the observation that a larger patch size results in lower performance. 2) Deeper joint modeling: utilizing a larger portion of the model (30%) for joint audio-visual processing, compared to <5% in CAV-MAE, enabling deeper cross-modal interaction without significantly increasing the computational cost associated with the concatenated audio-visual tokens.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 23 of its 26 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

23 of 26 references.

References · 23

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2024VMamba: Visual State Space Modelcited by 969
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  6. 2017Look, Listen and Learncited by 458
  7. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  8. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  9. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  10. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  11. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 60
  12. 2022MAViL: Masked Audio-Video Learnerscited by 59
11 more
  1. 2022Audiovisual Masked Autoencoderscited by 49
  2. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  3. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 37
  4. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 23
  5. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 16
  6. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  7. 2024EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learningcited by 12
  8. 2024Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localizationcited by 11
  9. 2024Audio Mamba: Pretrained Audio State Space Model For Audio Taggingcited by 10
  10. 2021Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicitycited by 8
  11. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learnerscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024State-Space Large Audio Language Models
  2. 2025SAM: A Mamba-2 State-Space Audio-Language Model
  3. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
  4. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time
  5. 2026Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
  6. 2026Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals ↗
  7. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  8. 2024Audio-visual training for improved grounding in video-text LLMs