mlsanityData to 7 Oct 2026

Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling

arXiv 2605.18281 ↗First posted 18 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How does such non-stationarity impact the inductive biases of deep learning towards models with different structural, generalisation, and safety properties? A fruitful testbed for studying inductive bias is in-context linear regression sequence modelling, where small transformers display strikingly different generalisation patterns depending on the diversity of the (fixed) training task distribution. In this paper, we explore the effect of diversifying the task distribution across training time, finding that such temporal diversity leads to an increased bias towards generalisation over memorisation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 50 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

19 of 50 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  8. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  9. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 90
  10. 2023In-Context Learning through the Bayesian Prismcited by 78
  11. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  12. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
7 more
  1. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 52
  2. 2021Martingale posterior distributionscited by 43
  3. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  4. 2019Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Taskscited by 39
  5. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  6. 2025In-Context Learning Strategies Emerge Rationallycited by 15
  7. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12

Built on by

Not cited yet.