mlsanityData to 14 Oct 2026

Trust-Region Twisted Policy Improvement

arXiv 2504.06048 ↗First posted 8 Apr 2025cs.LGcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Monte-Carlo tree search (MCTS) has driven many recent breakthroughs in deep reinforcement learning (RL). However, scaling MCTS to parallel compute has proven challenging in practice which has motivated alternative planners like sequential Monte-Carlo (SMC). Many of these SMC methods adopt particle filters for smoothing through a reformulation of RL as a policy inference problem. Yet, persisting design choices of these particle filters often conflict with the aim of online planning in RL, which is to obtain a policy improvement at the start of planning. Drawing inspiration from MCTS, we tailor SMC planners specifically for RL by improving data generation within the planner through constrained action sampling and explicit terminal state handling, as well as improving policy and value target estimation. This leads to our Trust-Region Twisted SMC (TRT-SMC), which shows improved runtime and sample-efficiency over baseline MCTS and SMC methods in both discrete and continuous domains.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 59 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

31 of 59 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  7. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  8. 2018Maximum a Posteriori Policy Optimisationcited by 328
  9. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  10. 2021Mastering Atari Games with Limited Datacited by 237
  11. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  12. 2017Variational Sequential Monte Carlocited by 95
19 more
  1. 2021Learning and Planning in Complex Action Spacescited by 77
  2. 2019An investigation of model-free planningcited by 66
  3. 2019Elements of Sequential Monte Carlocited by 64
  4. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  5. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
  6. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  7. 2023Empirical Design in Reinforcement Learningcited by 55
  8. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  9. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 40
  10. 2015Neural Adaptive Sequential Monte Carlocited by 34
  11. 2023Amortized Variational Inference: When and Why?cited by 29
  12. 2021Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Controlcited by 25
  13. 2022Critic Sequential Monte Carlocited by 16
  14. 2021Visualizing MuZero Modelscited by 9
  15. 2022Epistemic Monte Carlo Tree Searchcited by 7
  16. 2023What model does MuZero learn?cited by 7
  17. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 5
  18. 2015Nonlinear state space smoothing using the conditional particle filtercited by 3
  19. 2015Coarse-to-Fine Sequential Monte Carlo for Probabilistic Programscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scalingcited by 4
  2. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  3. 2026VariBASed: Variational Bayes-Adaptive Sequential Monte-Carlo Planning for Deep Reinforcement Learningcited by 1
  4. 2026Particle Monte Carlo Tree Searchcited by 0
  5. 2026Towards Optimal Policy Improvementcited by 0
  6. 2026Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFTcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Variance-Aware Prior-Based Tree Policies for Monte Carlo Tree Search
  2. 2024Enhancing Reinforcement Learning Through Guided Search
  3. 2026Online Robust Reinforcement Learning Through Monte-Carlo Planning
  4. 2024Optimized Monte Carlo Tree Search for Enhanced Decision Making in the FrozenLake Environment
  5. 2024Decision Making in Non-Stationary Environments with Policy-Augmented Search
  6. 2026Is Monte Carlo Tree Search Just Every-Visit Monte Carlo Control?
  7. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
  8. 2025TransZero: Parallel Tree Expansion in MuZero using Transformer Networks