# Generative sequence modeling for infinite memory processes via predictive states

arXiv: https://arxiv.org/abs/2609.38524 · HTML: https://mlsanity.com/paper/2609.38524/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 55 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem](https://mlsanity.com/paper/2306.06529/) (2023)
- [Hilbert Space Embeddings of Predictive State Representations](https://mlsanity.com/paper/1309.6819/) (2013)
- [Predictive State Recurrent Neural Networks](https://mlsanity.com/paper/1705.09353/) (2017)
- [Predictive-State Decoders: Encoding the Future into Recurrent Networks](https://mlsanity.com/paper/1709.08520/) (2017)
- [Testing for the Markov Property in Time Series via Deep Conditional Generative Learning](https://mlsanity.com/paper/2305.19244/) (2023)
- [Simulating and Reconstructing Neurodynamics with Epsilon-Automata Applied to Electroencephalography (EEG) Microstate Sequences](https://mlsanity.com/paper/1710.02387/) (2017)
- [Estimating dynamic models by matching random features](https://mlsanity.com/paper/2607.21916/) (2026)

## Abstract (arXiv)

We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.