# How Patterns Dictate Learnability in Sequential Data

arXiv: https://arxiv.org/abs/2510.10744 · HTML: https://mlsanity.com/paper/2510.10744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 12 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [Predictive Information Accelerates Learning in RL](https://mlsanity.com/paper/2007.12401/) (2020)
- [PAC-Bayes with Backprop](https://mlsanity.com/paper/1908.07380/) (2019)
- [TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series](https://mlsanity.com/paper/2310.01327/) (2023)
- [Learning Whenever Learning is Possible: Universal Learning under General Stochastic Processes](https://mlsanity.com/paper/1706.01418/) (2017)
- [Empirical Bayes Regret Minimization](https://mlsanity.com/paper/1904.02664/) (2019)
- [On Learnability under General Stochastic Processes](https://mlsanity.com/paper/2005.07605/) (2020)
- [Prospective Learning: Learning for a Dynamic Future](https://mlsanity.com/paper/2411.00109/) (2024)
- [Markov Chain Order estimation with Conditional Mutual Information](https://mlsanity.com/paper/1301.0148/) (2013)

## Abstract (arXiv)

Sequential data - ranging from financial time series to natural language - has driven the growing adoption of autoregressive models. However, these algorithms rely on the presence of underlying patterns in the data, and their identification often depends heavily on human expertise. Misinterpreting these patterns can lead to model misspecification, resulting in increased generalization error and degraded performance. The recently proposed evolving pattern (EvoRate) metric addresses this by using the mutual information between the next data point and its past to guide regression order estimation and feature selection. Building on this idea, we introduce a general framework based on predictive information, defined as the mutual information between the past and the future, I(Xpast; Xfuture). This quantity naturally defines an information-theoretic learning curve, which quantifies the amount of predictive information available as the observation window grows. Using this formalism, we show that the presence or absence of temporal patterns fundamentally constrains the learnability of sequential models: even an optimal predictor cannot outperform the intrinsic information limit imposed by the data. We validate our framework through experiments on synthetic data, demonstrating its ability to assess model adequacy, quantify the inherent complexity of a dataset, and reveal interpretable structure in sequential data.