mlsanityData to 7 Oct 2026

Structured State Space Decoder for Speech Recognition and Synthesis

arXiv 2210.17098 ↗First posted 31 Oct 2022cs.SD cs.LG eess.AScited by 11

On the map: State-space sequence models (S4, Mamba) → Speech and audio SSMs

Abstract (arXiv)

Automatic speech recognition (ASR) systems developed in recent years have shown promising results with self-attention models (e.g., Transformer and Conformer), which are replacing conventional recurrent neural networks. Meanwhile, a structured state space model (S4) has been recently proposed, producing promising results for various long-sequence modeling tasks, including raw speech classification. The S4 model can be trained in parallel, same as the Transformer model. In this study, we applied S4 as a decoder for ASR and text-to-speech (TTS) tasks by comparing it with the Transformer decoder. For the ASR task, our experimental results demonstrate that the proposed model achieves a competitive word error rate (WER) of 1.88%/4.25% on LibriSpeech test-clean/test-other set and a character error rate (CER) of 3.80%/2.63%/2.98% on the CSJ eval1/eval2/eval3 set. Furthermore, the proposed model is more robust than the standard Transformer model, particularly for long-form speech on both the datasets. For the TTS task, the proposed method outperforms the Transformer baseline.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 26 of its 36 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

26 of 36 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
14 more
  1. 2016Language Modeling with Gated Convolutional Networkscited by 688
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  4. 2015Listen, Attend and Spellcited by 542
  5. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  6. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  7. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  8. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  9. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  10. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  11. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  12. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  13. 2021A Comparative Study on Neural Architectures and Training Methods for Japanese Speech Recognitioncited by 6
  14. 2021SRU++: Pioneering Fast Recurrence with Attention for Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 3; 2026: 1.

11 papers.

  1. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  2. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  3. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  4. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  5. 2023A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Modelscited by 9
  6. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  7. 2023Augmenting conformers with structured state-space sequence models for online speech recognitioncited by 3
  8. 2025Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractionscited by 2
  9. 2025CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challengecited by 1
  10. 2026MagpieTTS-LF: Inference-Time Long-Form Speech Generation Without Training on Long-Form datacited by 1
  11. 2025Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Samba-ASR: State-Of-The-Art Speech Recognition Leveraging Structured State-Space Models
  2. 2023State Spaces Aren't Enough: Machine Translation Needs Attention
  3. 2023Multi-Head State Space Model for Speech Recognition
  4. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  5. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
  6. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  7. 2022Liquid Structural State-Space Models
  8. 2023Spiking Structured State Space Model for Monaural Speech Enhancement