mlsanityData to 7 Oct 2026

Near-Optimal Goal-Oriented Reinforcement Learning in Non-Stationary Environments

arXiv 2205.13044 ↗First posted 25 May 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We initiate the study of dynamic regret minimization for goal-oriented reinforcement learning modeled by a non-stationary stochastic shortest path problem with changing cost and transition functions. We start by establishing a lower bound Omega((Bstar SATstar(Deltac + Bstar2DeltaP))1/3K2/3), where Bstar is the maximum expected cost of the optimal policy of any episode starting from any state, Tstar is the maximum hitting time of the optimal policy of any episode starting from the initial state, SA is the number of state-action pairs, Deltac and DeltaP are the amount of changes of the cost and transition functions respectively, and K is the number of episodes. The different roles of Deltac and DeltaP in this lower bound inspire us to design algorithms that estimate costs and transitions separately. Specifically, assuming the knowledge of Deltac and DeltaP, we develop a simple but sub-optimal algorithm and another more involved minimax optimal algorithm (up to logarithmic terms). These algorithms combine the ideas of finite-horizon approximation [Chen et al., 2022a], special Bernstein-style bonuses of the MVP algorithm [Zhang et al., 2020], adaptive confidence widening [Wei and Luo, 2021], as well as some new techniques such as properly penalizing long-horizon policies. Finally, when Deltac and DeltaP are unknown, we develop a variant of the MASTER algorithm [Wei and Luo, 2021] and integrate the aforementioned ideas into it to achieve widetildeO(minBstar S√ALK, (Bstar2S2ATstar(Deltac+BstarDeltaP))1/3K2/3) regret, where L is the unknown number of changes of the environment.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

33 of 43 references.

References · 33

  1. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  2. 2020Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncited by 99
  3. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  4. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
  5. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 75
  6. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  7. 2019A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-freecited by 71
  8. 2020Near-optimal Regret Bounds for Stochastic Shortest Pathcited by 50
  9. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  10. 2019No-Regret Exploration in Goal-Oriented Reinforcement Learningcited by 34
  11. 2021Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regretcited by 34
  12. 2019Variational Regret Bounds for Reinforcement Learningcited by 32
21 more
  1. 2020Dynamic Regret of Policy Optimization in Non-stationary Environmentscited by 32
  2. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  3. 2021Minimax Regret for Stochastic Shortest Pathcited by 30
  4. 2018A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitionscited by 29
  5. 2020Stochastic Shortest Path with Adversarially Changing Costscited by 28
  6. 2021Finding the Stochastic Shortest Path with Low Regret: The Adversarial Cost and Unknown Transition Casecited by 28
  7. 2022A New Look at Dynamic Regret for Non-Stationary Stochastic Banditscited by 28
  8. 2020Minimax Regret for Stochastic Shortest Path with Adversarial Costs and Known Transitioncited by 27
  9. 2020Efficient Learning in Non-Stationary Linear Markov Decision Processescited by 26
  10. 2020Algorithms for Non-Stationary Generalized Linear Banditscited by 23
  11. 2020Nonstationary Reinforcement Learning with Linear Function Approximationcited by 23
  12. 2021Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Pathcited by 23
  13. 2021Learning Stochastic Shortest Path with Linear Function Approximationcited by 23
  14. 2016Refined Lower Bounds for Adversarial Banditscited by 20
  15. 2020Combinatorial Semi-Bandit in the Non-Stationary Environmentcited by 17
  16. 2021Online Learning for Stochastic Shortest Path Model via Posterior Samplingcited by 16
  17. 2021Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximationcited by 15
  18. 2021Improved No-Regret Algorithms for Stochastic Shortest Path with Linear MDPcited by 14
  19. 2021Regret Bounds for Generalized Linear Bandits under Parameter Driftcited by 13
  20. 2022Policy Optimization for Stochastic Shortest Pathcited by 13
  21. 2022Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraintscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 0; 2026: 1.

6 papers.

  1. 2023Tracking Most Significant Shifts in Nonparametric Contextual Banditscited by 10
  2. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  3. 2022Reaching Goals is Hard: Settling the Sample Complexity of the Stochastic Shortest Pathcited by 4
  4. 2022A Unified Algorithm for Stochastic Path Problemscited by 1
  5. 2026Achieving Optimal Static and Dynamic Regret Simultaneously in Bandits with Deterministic Lossescited by 1
  6. 2023Layered State Discovery for Incremental Autonomous Explorationcited by 0