mlsanityData to 7 Oct 2026

Tuning Frequency Bias of State Space Models

arXiv 2410.02035 ↗First posted 2 Oct 2024cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

State space models (SSMs) leverage linear, time-invariant (LTI) systems to effectively learn sequences with long-range dependencies. By analyzing the transfer functions of LTI systems, we find that SSMs exhibit an implicit bias toward capturing low-frequency components more effectively than high-frequency ones. This behavior aligns with the broader notion of frequency bias in deep learning model training. We show that the initialization of an SSM assigns it an innate frequency bias and that training the model in a conventional way does not alter this bias. Based on our theory, we propose two mechanisms to tune frequency bias: either by scaling the initialization to tune the inborn frequency bias; or by applying a Sobolev-norm-based filter to adjust the sensitivity of the gradients to high-frequency inputs, which allows us to change the frequency bias via training. Using an image-denoising task, we empirically show that we can strengthen, weaken, or even reverse the frequency bias using both mechanisms. By tuning the frequency bias, we can also improve SSMs' performance on learning long-range sequences, averaging an 88.26% accuracy on the Long-Range Arena (LRA) benchmark tasks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 49 of its 77 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

49 of 77 references.

References · 49

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  10. 2018On the Spectral Bias of Neural Networkscited by 1,321
  11. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  12. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
37 more
  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  4. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  7. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  9. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  10. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  11. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  12. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  13. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  14. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  15. 2017Sobolev Training for Neural Networkscited by 136
  16. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  17. 2022Liquid Structural State-Space Modelscited by 134
  18. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  19. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  20. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  21. 2020Lipschitz Recurrent Neural Networkscited by 88
  22. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  23. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  24. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  25. 2022Mitigating spectral bias for the multiscale operator learningcited by 43
  26. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  27. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  28. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 35
  29. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  30. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  31. 2023Spectral State Space Modelscited by 20
  32. 2023Convolutional State Space Models for Long-Range Spatiotemporal Modelingcited by 19
  33. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  34. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  35. 2020Understanding Recurrent Neural Networks Using Nonequilibrium Response Theorycited by 10
  36. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  37. 2024Towards a theory of learning dynamics in deep state space modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Binned Spectral Power Loss for Improved Prediction of Chaotic Systemscited by 10
  2. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  3. 2025A Deep State Space Model for Rainfall-Runoff Simulationscited by 3
  4. 2026Is Flow Matching Just Trajectory Replay for Sequential Data?cited by 2
  5. 2025MVHybrid: Improving Spatial Transcriptomics Prediction with Hybrid State Space-Vision Transformer Backbone in Pathology Vision Foundation Modelscited by 1
  6. 2025HydroDiffusion: Diffusion-Based Probabilistic Streamflow Forecasting with a State Space Backbonecited by 1
  7. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  8. 2025OLMA: One Loss for More Accurate Time Series Forecastingcited by 0
  9. 2025mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modelingcited by 0
  10. 2025FaRMamba: Frequency-based learning and Reconstruction aided Mamba for Medical Segmentationcited by 0
  11. 2025SpectMamba: Integrating Frequency and State Space Models for Enhanced Medical Image Detectioncited by 0
  12. 2025Aligning Inductive Bias for Data-Efficient Generalization in State Space Modelscited by 0
4 more
  1. 2025HPC-Driven Modeling with ML-Based Surrogates for Magnon-Photon Dynamics in Hybrid Quantum Systemscited by 0
  2. 2026Structured State-Space Regularization for Generation-Friendly Image Tokenizationcited by 0
  3. 2026Continuity Laws for Sequential Modelscited by 0
  4. 2026Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  2. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
  3. 2025Shaping Inductive Bias in Diffusion Models through Frequency-Based Noise Control
  4. 2025The Curious Case of In-Training Compression of State Space Models
  5. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels
  6. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
  7. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
  8. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systems