mlsanityData to 14 Oct 2026

Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning

arXiv 2508.16027 ↗First posted 22 Aug 2025stat.ML cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Transformers have demonstrated exceptional performance across a wide range of domains. While their ability to perform reinforcement learning in-context has been established both theoretically and empirically, their behavior in non-stationary environments remains less understood. In this study, we address this gap by showing that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings. We prove that transformers are capable of approximating strategies used to handle non-stationary environments and can learn the approximator in the in-context learning setup. Our experiments further show that transformers can match or even outperform existing expert algorithms in such environments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 47 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2019Graph Transformer Networkscited by 523
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  9. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  10. 2022Multi-Game Decision Transformerscited by 214
  11. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  12. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
18 more
  1. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  2. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  3. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  4. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  5. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  6. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  7. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  8. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  9. 2019A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-freecited by 71
  10. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  11. 2019Hedging the Drift: Learning to Optimize under Non-Stationaritycited by 46
  12. 2024Transformers are Universal In-context Learnerscited by 43
  13. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  14. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  15. 2019Randomized Exploration for Non-Stationary Stochastic Linear Banditscited by 18
  16. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  17. 2024Sliding-Window Thompson Sampling for Non-Stationary Settingscited by 17
  18. 2020Continuous-Time Multi-Armed Bandits with Controlled Restartscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  2. 2026Provable Transformer Context Compression for In-Context Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Prior-Free Black-Box Non-Stationary Reinforcement Learning Feasible?
  2. 2024MetaCURL: Non-stationary Concave Utility Reinforcement Learning
  3. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  4. 2023Transformers in Reinforcement Learning: A Survey
  5. 2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making
  6. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
  7. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  8. 2024Pausing Policy Learning in Non-stationary Reinforcement Learning