# Long-Context Linear System Identification

arXiv: https://arxiv.org/abs/2410.05690 · HTML: https://mlsanity.com/paper/2410.05690/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification](https://mlsanity.com/paper/1802.08334/) (2018)
- [Scalable Pre-training of Large Autoregressive Image Models](https://mlsanity.com/paper/2401.08541/) (2024)
- [Near optimal finite time identification of arbitrary linear dynamical systems](https://mlsanity.com/paper/1812.01251/) (2018)
- [Active Learning for Nonlinear System Identification with Guarantees](https://mlsanity.com/paper/2006.10277/) (2020)
- [Gradient Descent Learns Linear Dynamical Systems](https://mlsanity.com/paper/1609.05191/) (2016)
- [Learning Dynamical Systems via Koopman Operator Regression in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/2205.14027/) (2022)
- [Finite Time Identification in Unstable Linear Systems](https://mlsanity.com/paper/1710.01852/) (2017)
- [Statistical Learning Theory for Control: A Finite Sample Perspective](https://mlsanity.com/paper/2209.05423/) (2022)
- [Learning Linear Dynamical Systems via Spectral Filtering](https://mlsanity.com/paper/1711.00946/) (2017)
- [Learning nonlinear dynamical systems from a single trajectory](https://mlsanity.com/paper/2004.14681/) (2020)
- [Stochastic Gradient Descent under Markovian Sampling Schemes](https://mlsanity.com/paper/2302.14428/) (2023)
- [Learning with little mixing](https://mlsanity.com/paper/2206.08269/) (2022)
- [MAML and ANIL Provably Learn Representations](https://mlsanity.com/paper/2202.03483/) (2022)
- [Learning from many trajectories](https://mlsanity.com/paper/2203.17193/) (2022)
- [Sample Complexity Lower Bounds for Linear System Identification](https://mlsanity.com/paper/1903.10343/) (2019)
- [Streaming Linear System Identification with Reverse Experience Replay](https://mlsanity.com/paper/2103.05896/) (2021)
- [Low Rank and Structured Modeling of High-dimensional Vector Autoregressions](https://mlsanity.com/paper/1812.03568/) (2018)
- [Trace norm regularization for multi-task learning with scarce data](https://mlsanity.com/paper/2202.06742/) (2022)
- [High dimensional VAR with low rank transition](https://mlsanity.com/paper/1905.00959/) (2019)
- [Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processes](https://mlsanity.com/paper/1404.6769/) (2014)
- [Autoregressive Bandits](https://mlsanity.com/paper/2212.06251/) (2022)

## Built on by (most cited first, 3 of 3)

- [In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation](https://mlsanity.com/paper/2502.08136/) (2025)
- [Incremental Learning of Sparse Attention Patterns in Transformers](https://mlsanity.com/paper/2602.19143/) (2026)
- [Induction Heads Interpolate N-Grams](https://mlsanity.com/paper/2607.02800/) (2026)

## Abstract (arXiv)

This paper addresses the problem of long-context linear system identification, where the state xt of a dynamical system at time t depends linearly on previous states xs over a fixed context window of length p. We establish a sample complexity bound that matches the i.i.d. parametric rate up to logarithmic factors for a broad class of systems, extending previous works that considered only first-order dependencies. Our findings reveal a learning-without-mixing phenomenon, indicating that learning long-context linear autoregressive models is not hindered by slow mixing properties potentially associated with extended context windows. Additionally, we extend these results to (i) shared low-rank representations, where rank-regularized estimators improve the dependence of the rates on the dimensionality, and (ii) misspecified context lengths in strictly stable systems, where shorter contexts offer statistical advantages.