# DAVSS: Distilled Audio-Visual State Space Models

arXiv: https://arxiv.org/abs/2608.19523 · HTML: https://mlsanity.com/paper/2608.19523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 23 of 23)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [Audio Mamba: Bidirectional State Space Model for Audio Representation Learning](https://mlsanity.com/paper/2406.03344/) (2024)
- [Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis](https://mlsanity.com/paper/2407.09732/) (2024)
- [CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment](https://mlsanity.com/paper/2505.01237/) (2025)
- [Siamese Vision Transformers are Scalable Audio-visual Learners](https://mlsanity.com/paper/2403.19638/) (2024)
- [EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning](https://mlsanity.com/paper/2403.09502/) (2024)
- [Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization](https://mlsanity.com/paper/2409.07967/) (2024)
- [Audio Mamba: Pretrained Audio State Space Model For Audio Tagging](https://mlsanity.com/paper/2405.13636/) (2024)
- [Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity](https://mlsanity.com/paper/2111.05329/) (2021)
- [DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners](https://mlsanity.com/paper/2407.04082/) (2024)

## Abstract (arXiv)

State-space models (SSMs) distilled from transformer teachers combine the performance of transformers with the efficiency of SSMs. We extend the Transformer-SSM knowledge distillation to a multimodal setting and propose the Distilled Audio-visual State-Space (DAVSS) model. The DAVSS model, 14M parameters, is 12 times smaller compared to transformer-based models such as CAV-MAE, and still outperforms them. DAVSS improves over the existing audio-visual models by: 1) Finer input resolution: using smaller patch sizes process the input, compensating for the smaller model size by increasing input sequence lengths. This is supported by the observation that a larger patch size results in lower performance. 2) Deeper joint modeling: utilizing a larger portion of the model (30%) for joint audio-visual processing, compared to <5% in CAV-MAE, enabling deeper cross-modal interaction without significantly increasing the computational cost associated with the concatenated audio-visual tokens.