mlsanityData to 14 Oct 2026

Flash STU: Fast Spectral Transform Units

arXiv 2409.10489 ↗First posted 16 Sep 2024cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances in state-space model architectures have shown great promise for efficient sequence modeling, but challenges remain in balancing computational efficiency with model expressiveness. We propose the Flash STU architecture, a hybrid model that interleaves spectral state space model layers with sliding window attention, enabling scalability to billions of parameters for language modeling while maintaining a near-linear time complexity. We evaluate the Flash STU and its variants on diverse sequence prediction tasks, including linear dynamical systems, robotics control, and language modeling. We find that, given a fixed parameter budget, the Flash STU architecture consistently outperforms the Transformer and other leading state-space models such as S4 and Mamba-2.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

47 of 74 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2020Longformer: The Long-Document Transformercited by 3,413
35 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  14. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  15. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  16. 2022In-context Learning and Induction Headscited by 964
  17. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  18. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  19. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  20. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
  21. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  22. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  23. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  24. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  25. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  26. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  27. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  28. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  29. 2016Gradient Descent Learns Linear Dynamical Systemscited by 79
  30. 2017Learning Linear Dynamical Systems via Spectral Filteringcited by 46
  31. 2018Spectral Filtering for General Linear Dynamical Systemscited by 37
  32. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  33. 2023Spectral State Space Modelscited by 21
  34. 2018On-Line Learning of Linear Dynamical Systems: Exponential Forgetting in Kalman Filterscited by 16
  35. 2023Position Interpolation Improves ALiBi Extrapolationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 4.

7 papers.

  1. 2024FutureFill: Fast Generation from Convolutional Sequence Modelscited by 7
  2. 2025Universal Learning of Nonlinear Dynamicscited by 5
  3. 2024Provable Length Generalization in Sequence Prediction via Spectral Filteringcited by 3
  4. 2026SFO: Learning PDE Operators via Spectral Filteringcited by 0
  5. 2026RubiConv -- Efficient Boundary-Respecting Convolutionscited by 0
  6. 2026Spectral Distillation: From Nonlinear Dynamics to Linear State-Space Modelscited by 0
  7. 2026Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  2. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  3. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  4. 2023Block-State Transformers
  5. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modeling
  6. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
  7. 2024Q-S5: Towards Quantized State Space Models
  8. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling