mlsanityData to 7 Oct 2026

Dual Policy Iteration

arXiv 1805.10755 ↗First posted 28 May 2018cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved with guidance from the reactive policy. In this work we study this Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 2; 2021: 6; 2022: 2; 2023: 3; 2024: 2; 2025: 4; 2026: 1.

22 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  3. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 77
  4. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  5. 2019Interactive Teaching Algorithms for Inverse Reinforcement Learningcited by 24
  6. 2023Learning Energy Decompositions for Partial Inference of GFlowNetscited by 24
  7. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  8. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  9. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  10. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  11. 2019On-Policy Robot Imitation Learning from a Converging Supervisorcited by 10
  12. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
10 more
  1. 2022Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learningcited by 5
  2. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  3. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 4
  4. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  5. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  6. 2021Federated Ensemble Model-based Reinforcement Learning in Edge Computingcited by 2
  7. 2025Distributionally Robust Imitation Learning: Layered Control Architecture for Certifiable Autonomycited by 1
  8. 2021PEARL: Parallelized Expert-Assisted Reinforcement Learning for Scene Rearrangement Planningcited by 0
  9. 2021Reinforced Imitation Learning by Free Energy Principlecited by 0
  10. 2024Extremum-Seeking Action Selection for Accelerating Policy Optimizationcited by 0