Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Multiple-step lookahead policies have demonstrated high empirical competence in Reinforcement Learning, via the use of Monte Carlo Tree Search or Model Predictive Control. In a recent work citeefroni2018beyond, multiple-step greedy policies and their use in vanilla Policy Iteration algorithms were proposed and analyzed. In this work, we study multiple-step greedy algorithms in more practical setups. We begin by highlighting a counter-intuitive difficulty, arising with soft-policy updates: even in the absence of approximations, and contrary to the 1-step-greedy case, monotonic policy improvement is not guaranteed unless the update stepsize is sufficiently large. Taking particular care about this difficulty, we formulate and analyze online and approximate algorithms that use such a multi-step greedy operator.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 3; 2023: 3.
7 papers.
- 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 11
- 2022Planning and Learning with Adaptive Lookaheadcited by 8
- 2023On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluationcited by 8
- 2021The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximationcited by 5
- 2022Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximationcited by 4
- 2023A New Policy Iteration Algorithm For Reinforcement Learning in Zero-Sum Markov Gamescited by 0
- 2023Acceleration in Policy Optimizationcited by 0
- 2023Acceleration in Policy Optimization18 Jun 2023
- 2023A New Policy Iteration Algorithm For Reinforcement Learning in Zero-Sum Markov Games17 Mar 2023
- 2023On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation23 Jan 2023
- 2022Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximation13 Oct 2022
- 2022Generalised Policy Improvement with Geometric Policy Composition17 Jun 2022
- 2022Planning and Learning with Adaptive Lookahead28 Jan 2022
- 2021The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximation28 Sep 2021