mlsanityData to 14 Oct 2026

Long-Context Linear System Identification

arXiv 2410.05690 ↗First posted 8 Oct 2024stat.ML cs.LG cs.SYcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper addresses the problem of long-context linear system identification, where the state xt of a dynamical system at time t depends linearly on previous states xs over a fixed context window of length p. We establish a sample complexity bound that matches the i.i.d. parametric rate up to logarithmic factors for a broad class of systems, extending previous works that considered only first-order dependencies. Our findings reveal a learning-without-mixing phenomenon, indicating that learning long-context linear autoregressive models is not hindered by slow mixing properties potentially associated with extended context windows. Additionally, we extend these results to (i) shared low-rank representations, where rank-regularized estimators improve the dependence of the rates on the dimensionality, and (ii) misspecified context lengths in strictly stable systems, where shorter contexts offer statistical advantages.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

26 of 52 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  5. 2020Provable Meta-Learning of Linear Representationscited by 161
  6. 2018Learning Without Mixing: Towards A Sharp Analysis of Linear System Identificationcited by 158
  7. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  8. 2018Near optimal finite time identification of arbitrary linear dynamical systemscited by 89
  9. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 80
  10. 2016Gradient Descent Learns Linear Dynamical Systemscited by 79
  11. 2022Learning Dynamical Systems via Koopman Operator Regression in Reproducing Kernel Hilbert Spacescited by 67
  12. 2017Finite Time Identification in Unstable Linear Systemscited by 66
14 more
  1. 2022Statistical Learning Theory for Control: A Finite Sample Perspectivecited by 53
  2. 2017Learning Linear Dynamical Systems via Spectral Filteringcited by 46
  3. 2020Learning nonlinear dynamical systems from a single trajectorycited by 46
  4. 2023Stochastic Gradient Descent under Markovian Sampling Schemescited by 38
  5. 2022Learning with little mixingcited by 33
  6. 2022MAML and ANIL Provably Learn Representationscited by 30
  7. 2022Learning from many trajectoriescited by 24
  8. 2019Sample Complexity Lower Bounds for Linear System Identificationcited by 20
  9. 2021Streaming Linear System Identification with Reverse Experience Replaycited by 17
  10. 2018Low Rank and Structured Modeling of High-dimensional Vector Autoregressionscited by 12
  11. 2022Trace norm regularization for multi-task learning with scarce datacited by 10
  12. 2019High dimensional VAR with low rank transitioncited by 4
  13. 2014Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processescited by 3
  14. 2022Autoregressive Banditscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separationcited by 4
  2. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 2
  3. 2026Induction Heads Interpolate N-Gramscited by 1