mlsanityData to 7 Oct 2026

Optimistic Planning by Regularized Dynamic Programming

arXiv 2302.14004 ↗First posted 27 Feb 2023cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a new method for optimistic planning in infinite-horizon discounted Markov decision processes based on the idea of adding regularization to the updates of an otherwise standard approximate value iteration procedure. This technique allows us to avoid contraction and monotonicity arguments typically required by existing analyses of approximate dynamic programming methods, and in particular to use approximate transition functions estimated via least-squares procedures in MDPs with linear function approximation. We use our method to recover known guarantees in tabular MDPs and to provide a computationally efficient algorithm for learning near-optimal policies in discounted linear mixture MDPs from a single stream of experience, and show it achieves near-optimal statistical guarantees.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 16 of its 40 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

16 of 40 references.

References · 16

  1. 2019A Theory of Regularized Markov Decision Processescited by 279
  2. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  3. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  4. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  5. 2017A unified view of entropy-regularized Markov decision processescited by 174
  6. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  7. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  8. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  9. 2018Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learningcited by 65
  10. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  11. 2020Learning Infinite-horizon Average-reward MDPs with Linear Function Approximationcited by 48
  12. 2020Nearly Minimax Optimal Reinforcement Learning for Discounted MDPscited by 42
4 more
  1. 2020Regret Bounds for Discounted MDPscited by 19
  2. 2021Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximationcited by 15
  3. 2022Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration and Planningcited by 12
  4. 2018Exploration Bonus for Regret Minimization in Undiscounted Discrete and Continuous Markov Decision Processescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 1.

8 papers.

  1. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  2. 2024Operator World Models for Reinforcement Learningcited by 7
  3. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  4. 2024Bisimulation Metrics are Optimal Transport Distances, and Can be Computed Efficientlycited by 6
  5. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  6. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPscited by 5
  7. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 3
  8. 2026Value Mirror Descent for Reinforcement Learningcited by 1