mlsanityData to 7 Oct 2026

It's Raw! Audio Generation with State-Space Models

arXiv 2202.09729 ↗First posted 20 Feb 2022cs.SD cs.AI cs.LGcited by 171

Not on the map. Why?

Cited by 46 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Developing architectures suitable for modeling raw audio is a challenging problem due to the high sampling rates of audio waveforms. Standard sequence modeling approaches like RNNs and CNNs have previously been tailored to fit the demands of audio, but the resultant architectures make undesirable computational tradeoffs and struggle to model waveforms effectively. We propose SaShiMi, a new multi-scale architecture for waveform modeling built around the recently introduced S4 model for long sequence modeling. We identify that S4 can be unstable during autoregressive generation, and provide a simple improvement to its parameterization by drawing connections to Hurwitz matrices. SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting. Additionally, SaShiMi improves non-autoregressive generation performance when used as the backbone architecture for a diffusion model. Compared to prior architectures in the autoregressive generation setting, SaShiMi generates piano and speech waveforms which humans find more musical and coherent respectively, e.g. 2x better mean opinion scores than WaveNet on an unconditional speech generation task. On a music generation task, SaShiMi outperforms WaveNet on density estimation and speed at both training and inference even when using 3x fewer parameters. Code can be found at https://github.com/HazyResearch/state-spaces and samples at https://hazyresearch.stanford.edu/sashimi-examples.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 20 Feb 2022.

Builds on

38 of 55 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
26 more
  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  2. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  3. 2016Language Modeling with Gated Convolutional Networkscited by 688
  4. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2020Jukebox: A Generative Model for Musiccited by 535
  7. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  8. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  9. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  11. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  12. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  13. 2018Adversarial Audio Synthesiscited by 222
  14. 2018Efficient Neural Audio Synthesiscited by 218
  15. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  16. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  17. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  18. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  19. 2017DeLiGAN : Generative Adversarial Networks for Diverse and Limited Datacited by 77
  20. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  21. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  22. 2018FloWaveNet : A Generative Flow for Raw Audiocited by 54
  23. 2019WaveFlow: A Compact Flow-based Model for Raw Audiocited by 51
  24. 2017Wavenet based low rate speech codingcited by 32
  25. 2021Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamicscited by 19
  26. 2019Expediting TTS Synthesis with Adversarial Vocodingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 40; 2024: 66; 2025: 29; 2026: 9.

171 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  6. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  8. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  9. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  10. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  11. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  12. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
38 more
  1. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  2. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 95
  3. 2023Score-based Data Assimilationcited by 91
  4. 2024The Hidden Attention of Mamba Modelscited by 88
  5. 2023Diffusion Models Without Attentioncited by 75
  6. 2024State Space Models for Event Camerascited by 75
  7. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  8. 2023Diffusion Models for Time Series Applications: A Surveycited by 72
  9. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 66
  10. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  11. 2023Functional Flow Matchingcited by 63
  12. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  13. 2022Solving Audio Inverse Problems with a Diffusion Modelcited by 62
  14. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  15. 2023PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputationcited by 62
  16. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  17. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  18. 2023SingSong: Generating musical accompaniments from singingcited by 54
  19. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  20. 2023Spatio-temporal Diffusion Point Processescited by 52
  21. 2022Pretraining Without Attentioncited by 51
  22. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  23. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  24. 2024Is Mamba Capable of In-Context Learning?cited by 45
  25. 2025MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classificationcited by 44
  26. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  27. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  28. 2024Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Seriescited by 42
  29. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  30. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  31. 2024U-shaped Vision Mamba for Single Image Dehazingcited by 40
  32. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  33. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shapingcited by 37
  34. 2022Deep Latent State Space Models for Time-Series Generationcited by 37
  35. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  36. 2024Were RNNs All We Needed?cited by 36
  37. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 35
  38. 2024Locating and Editing Factual Associations in Mambacited by 35

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
  2. 2024Efficient Autoregressive Audio Modeling via Next-Scale Prediction
  3. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  4. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
  5. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation
  6. 2023ArchiSound: Audio Generation with Diffusion
  7. 2024Long-form music generation with latent diffusion
  8. 2025ACE-Step: A Step Towards Music Generation Foundation Model