# Prediction Is Not a Sufficient Memory State

arXiv: https://arxiv.org/abs/2610.13881 · HTML: https://mlsanity.com/paper/2610.13881/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 17 of 17)

- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Learning with Fenchel-Young Losses](https://mlsanity.com/paper/1901.02324/) (2019)
- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [On Coresets for Logistic Regression](https://mlsanity.com/paper/1805.08571/) (2018)
- [MesaNet: Sequence Modeling by Locally Optimal Test-Time Training](https://mlsanity.com/paper/2506.05233/) (2025)
- [Omnipredictors for Regression and the Approximate Rank of Convex Functions](https://mlsanity.com/paper/2401.14645/) (2024)
- [Online Learning: Sufficient Statistics and the Burkholder Method](https://mlsanity.com/paper/1803.07617/) (2018)
- [PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference](https://mlsanity.com/paper/1709.09216/) (2017)
- [Strong Memory Lower Bounds for Learning Natural Models](https://mlsanity.com/paper/2206.04743/) (2022)
- [Streaming Attention Approximation via Discrepancy Theory](https://mlsanity.com/paper/2502.07861/) (2025)
- [Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences](https://mlsanity.com/paper/2604.21100/) (2026)
- [Equivalence of Context and Parameter Updates in Modern Transformer Blocks](https://mlsanity.com/paper/2511.17864/) (2025)
- [Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models](https://mlsanity.com/paper/2603.08859/) (2026)
- [Gibbs Manifolds](https://mlsanity.com/paper/2211.15490/) (2022)
- [Learning to Remember, Learn, and Forget in Attention-Based Models](https://mlsanity.com/paper/2602.09075/) (2026)
- [Towards Tight Bounds for Streaming Attention](https://mlsanity.com/paper/2606.07205/) (2026)

## Abstract (arXiv)

A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. With ordinary, unweighted observations, degree D requires precisely r=binomd+DD-1 coordinates; even among histories with one fixed current predictor, exactly r-d coordinates can remain necessary for continuation. Nonpolynomial losses have unbounded requirements on such equal-prediction histories. Bounded scalar logistic examples retain this obstruction after fixing any finite derivative jet. For ridge regression, the hidden evidence is the design matrix: the exact state costs dp+d(d+1)/2, versus dp for the predictor. On a well-conditioned zero-prediction family, optimal B-bit compression has average one-update error of order d-12-2B/sd, sd=d(d+1)/2, even with arbitrary randomized codes. Kernel attention gives a second exact closure classification. The results identify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains.