# The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models

arXiv: https://arxiv.org/abs/2610.08816 · HTML: https://mlsanity.com/paper/2610.08816/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024)
- [LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement](https://mlsanity.com/paper/2504.16053/) (2025)
- [State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory](https://mlsanity.com/paper/2309.13414/) (2023)
- [StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization](https://mlsanity.com/paper/2311.14495/) (2023)
- [On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis](https://mlsanity.com/paper/2009.07799/) (2020)
- [Tuning Frequency Bias of State Space Models](https://mlsanity.com/paper/2410.02035/) (2024)
- [High-dimensional vector autoregressive time series modeling via tensor decomposition](https://mlsanity.com/paper/1909.06624/) (2019)
- [HOPE for a Robust Parameterization of Long-memory State Space Models](https://mlsanity.com/paper/2405.13975/) (2024)
- [Inverse Approximation Theory for Nonlinear Recurrent Neural Networks](https://mlsanity.com/paper/2305.19190/) (2023)
- [Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models](https://mlsanity.com/paper/2411.19455/) (2024)
- [L2M: Mutual Information Scaling Law for Long-Context Language Modeling](https://mlsanity.com/paper/2503.04725/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Supervised Factor Modeling for High-Dimensional Linear Time Series](https://mlsanity.com/paper/2312.00346/) (2023)
- [VORT: Adaptive Power-Law Memory for NLP Transformers](https://mlsanity.com/paper/2605.08966/) (2026)

## Abstract (arXiv)

Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately? We study this question directly in forecasting risk. For algebraically decaying predictive memory, we prove matching upper and lower approximation bounds for exponential and finite-state modes. The best r-mode forecast error decays as e-Theta(√ r), so reaching forecast error τ needs r=Theta(log2(1/τ)) states or modes. Earlier curse-of-memory results establish broad limitations of stable recurrent models under different approximation notions; here both sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target. We then show that genuine fractional long memory changes the geometry itself. In particular, forecast error is measured after fractional integration, prediction from a finite context of length L has an exact 1/L leading order, and a fixed fractional strength d keeps the square-log state-complexity law. Near the short-memory boundary, we identify the relevant d2 and d4 scales and give a uniform constructive law in the intermediate regime. For nonlinear contextual recurrences with uniformly contractive state dynamics, we derive an exponential first-chaos envelope and an explicit necessary condition that relates forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality, a condition that is necessary and not by itself sufficient. Finite-sample Kullback--Leibler calculations further connect the predictive geometry to statistical information. Theorem-matched experiments with contractive state-space, gated recurrent, and attention models reproduce the state and stability predictions.