mlsanityData to 14 Oct 2026

Stagewise Reinforcement Learning and the Geometry of the Regret Landscape

arXiv 2601.07524 ↗First posted 12 Jan 2026cs.LGcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Singular learning theory characterizes Bayesian learning as an evolving tradeoff between accuracy and complexity, with transitions between qualitatively different solutions as sample size increases. We extend this theory to reinforcement learning, proving that the concentration of a generalized posterior over policies is governed by the local learning coefficient (LLC), an invariant of the geometry of the regret function. This theory predicts that deep reinforcement learning with SGD should proceed from simple policies with high regret to complex policies with low regret. We verify this prediction empirically in a gridworld environment exhibiting stagewise policy development: phase transitions over training manifest as "opposing staircases" where regret decreases sharply while the LLC increases.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 77 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

43 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learning28 Jan

Other references · 42

  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2017Exploring Generalization in Deep Learningcited by 600
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  6. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  7. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  9. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  10. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  11. 2013A General Framework for Updating Belief Distributionscited by 175
  12. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
30 more
  1. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  2. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  3. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  4. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  6. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  7. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  8. 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
  9. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  10. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  11. 2020Deep Learning is Singular, and That's Goodcited by 49
  12. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  13. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  14. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  15. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  16. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  17. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  18. 2023Understanding and Controlling a Maze-Solving Policy Networkcited by 22
  19. 2025You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisationcited by 17
  20. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  21. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  22. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  23. 2025Structural Inference: Interpreting Small Language Models with Susceptibilitiescited by 13
  24. 2019Logic and the 2-Simplicial Transformercited by 9
  25. 2022Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environmentscited by 9
  26. 2023Model-agnostic Measure of Generalization Difficultycited by 7
  27. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  28. 2024A Bayesian Model Selection Criterion for Selecting Pretraining Checkpointscited by 5
  29. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  30. 2023Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Studycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Patterning: The Dual of Interpretabilitycited by 7
  2. 2026Interpreting Reinforcement Learning Agents with Susceptibilitiescited by 1
  3. 2026On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspectivecited by 0
  4. 2026Structure and Scale in Simplicial Sequence Modellingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs
  2. 2025The Geometry of Nonlinear Reinforcement Learning
  3. 2024Refining Minimax Regret for Unsupervised Environment Design
  4. 2025Asymptotically optimal reinforcement learning in Block Markov Decision Processes
  5. 2025Rethinking the Foundations for Continual Reinforcement Learning
  6. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
  7. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward
  8. 2025Regret Bounds for Robust Online Decision Making