mlsanityData to 14 Oct 2026

Multi-Head State Space Model for Speech Recognition

arXiv 2305.12498 ↗First posted 21 May 2023eess.AS cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

State space models (SSMs) have recently shown promising results on small-scale sequence and language modelling tasks, rivalling and outperforming many attention-based approaches. In this paper, we propose a multi-head state space (MH-SSM) architecture equipped with special gating mechanisms, where parallel heads are taught to learn local and global temporal dynamics on sequence data. As a drop-in replacement for multi-head attention in transformer encoders, this new model significantly outperforms the transformer transducer on the LibriSpeech speech recognition corpus. Furthermore, we augment the transformer block with MH-SSMs layers, referred to as the Stateformer, achieving state-of-the-art performance on the LibriSpeech task, with word error rates of 1.76%/4.37% on the development and 1.91%/4.36% on the test sets without using an external language model.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
20 more
  1. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  4. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  5. 2015Listen, Attend and Spellcited by 543
  6. 2022Long Range Language Modeling via Gated State Spacescited by 265
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  8. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  9. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  10. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  11. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  12. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
  13. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  14. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  15. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 88
  16. 2020Talking-Heads Attentioncited by 62
  17. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 61
  18. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  19. 2020Improving RNN Transducer Based ASR with Auxiliary Taskscited by 27
  20. 2021Parallelizing Legendre Memory Unit Trainingcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  2. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 44
  3. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  4. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  5. 2024Exploring the Capability of Mamba in Speech Applicationscited by 20
  6. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 17
  7. 2023On the Long Range Abilities of Transformerscited by 10
  8. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  9. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4
  10. 2023Augmenting conformers with structured state-space sequence models for online speech recognitioncited by 3
  11. 2025TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embeddingcited by 3
  12. 2025MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancementcited by 2
2 more
  1. 2025Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASRcited by 0
  2. 2026FoSS: Modeling Long Range Dependencies and Multimodal Uncertainty in Trajectory Prediction via Fourier State Space Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Block-State Transformers
  2. 2024State-Space Large Audio Language Models
  3. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  4. 2022Structured State Space Decoder for Speech Recognition and Synthesis
  5. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
  7. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  8. 2026WaveSSM: Multiscale State-Space Models for Non-stationary Signal Attention