# In-Context Learning Under Regime Change

arXiv: https://arxiv.org/abs/2604.16988 · HTML: https://mlsanity.com/paper/2604.16988/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization](https://mlsanity.com/paper/2305.19420/) (2023)
- [From Tables to Time: Extending TabPFN-v2 to Time Series Forecasting](https://mlsanity.com/paper/2501.02945/) (2025)
- [TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Data](https://mlsanity.com/paper/2502.16294/) (2025)
- [Gating is Weighting: Understanding Gated Linear Attention through In-context Learning](https://mlsanity.com/paper/2504.04308/) (2025)
- [Simulation as Supervision: Mechanistic Pretraining for Scientific Discovery](https://mlsanity.com/paper/2507.08977/) (2025)
- [Mantis: A Foundation Model for Mechanistic Disease Forecasting](https://mlsanity.com/paper/2508.12260/) (2025)
- [Learning From Simulators: A Theory of Simulation-Grounded Learning](https://mlsanity.com/paper/2509.18990/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Should an In-Context Learner Expand Its Hypothesis Space?](https://mlsanity.com/paper/2610.09471/) (2026)

## Abstract (arXiv)

Non-stationary sequences arise naturally in control, forecasting, and decision-making. The data-generating process shifts at unknown times, and models must detect the change, discard or downweight obsolete evidence, and adapt to new dynamics on the fly. Transformer-based foundation models increasingly rely on in-context learning for time series forecasting, tabular prediction, and continuous control. As these models are deployed in non-stationary environments, understanding their ability to detect and adapt to regime shifts is important. We formalize this as an in-context change-point detection problem and formally establish the existence of transformer models that solve this problem. Our construction demonstrates that model complexity, in layers and parameters, depends on the level of information available about the change-point location, from no knowledge to knowing exact timing. We validate our results with experiments on synthetic linear regression and linear dynamical systems, where trained transformers match the performance of optimal baselines across information levels. We also show that encoding and incorporating changepoint knowledge indeed improves the real-world performance of a pretrained foundation models on infectious disease forecasting and on financial volatility forecasting around Federal Open Market Committee (FOMC) announcements without retraining, demonstrating practical applicability to real-world regime changes.