mlsanityData to 13 Oct 2026

Beyond the One Step Greedy Approach in Reinforcement Learning

arXiv 1802.03654 ↗First posted 10 Feb 2018cs.AI cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

The famous Policy Iteration algorithm alternates between policy improvement and policy evaluation. Implementations of this algorithm with several variants of the latter evaluation stage, e.g, n-step and trace-based returns, have been analyzed in previous works. However, the case of multiple-step lookahead policy improvement, despite the recent increase in empirical evidence of its strength, has to our knowledge not been carefully analyzed yet. In this work, we introduce the first such analysis. Namely, we formulate variants of multiple-step policy improvement, derive new algorithms using these definitions and prove their convergence. Moreover, we show that recent prominent Reinforcement Learning algorithms are, in fact, instances of our framework. We thus shed light on their empirical success and give a recipe for deriving new algorithms for future study.

Key facts. Cited by 20 papers in the map (data through 13 Oct 2026). First posted on arXiv 10 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 5; 2024: 2; 2025: 5; 2026: 1.

20 papers.

Sort:

By area:

  1. 2021Heuristic-Guided Reinforcement Learningcited by 41
  2. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 11
  3. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10
  4. 2022Planning and Learning with Adaptive Lookaheadcited by 8
  5. 2023On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluationcited by 8
  6. 2024Policy Mirror Descent with Lookaheadcited by 6
  7. 2021The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximationcited by 5
  8. 2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 4
  9. 2023Policy Gradient with Tree Expansioncited by 3
  10. 2023Multi-Bellman operator for convergence of Q-learning with linear function approximationcited by 3
  11. 2025Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approachcited by 3
  12. 2024Highway Reinforcement Learningcited by 2
8 more
  1. 2025Counting Hours, Counting Losses: The Toll of Unpredictable Work Schedules on Financial Securitycited by 2
  2. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookaheadcited by 1
  3. 2025Monotone and Conservative Policy Iteration Beyond the Tabular Casecited by 1
  4. 2021Greedy-Step Off-Policy Reinforcement Learningcited by 0
  5. 2021Learning to Plan via a Multi-Step Policy Regression Methodcited by 0
  6. 2023On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processescited by 0
  7. 2023A New Policy Iteration Algorithm For Reinforcement Learning in Zero-Sum Markov Gamescited by 0
  8. 2026Partial Policy Gradients for RL in LLMscited by 0