# HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit

arXiv: https://arxiv.org/abs/2603.21316 · HTML: https://mlsanity.com/paper/2603.21316/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 15 of 15)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model](https://mlsanity.com/paper/2405.11831/) (2024)
- [Audio Mamba: Bidirectional State Space Model for Audio Representation Learning](https://mlsanity.com/paper/2406.03344/) (2024)

## Abstract (arXiv)

Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure attention, and a minimal hybrid with a single attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six datasets, we find that the preferred input representation depends on the backbone, and that attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000 tokens, pure attention fails with out-of-memory errors, while HELIX closes an 11.5-point gap over pure Mamba.