mlsanityData to 14 Oct 2026

Twice Sequential Monte Carlo for Tree Search

arXiv 2511.14220 ↗First posted 18 Nov 2025cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Model-based reinforcement learning (RL) methods that leverage search are responsible for many milestone breakthroughs in RL. Sequential Monte Carlo (SMC) recently emerged as an alternative to the Monte Carlo Tree Search (MCTS) algorithm which drove these breakthroughs. SMC is easier to parallelize and more suitable to GPU acceleration. However, it also suffers from large variance and path degeneracy which prevent it from scaling well with increased search depth, i.e., increased sequential compute. To address these problems, we introduce Twice Sequential Monte Carlo Tree Search (TSMCTS). Across discrete and continuous environments TSMCTS outperforms the SMC baseline as well as a popular modern version of MCTS as a policy improvement operator, scales favorably with sequential compute, reduces estimator variance and mitigates the effects of path degeneracy while retaining the properties that make SMC natural to parallelize.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 46 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

20 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Trust-Region Twisted Policy Improvement8 Apr
  2. 2024Value Improved Actor Critic Algorithms3 Jun

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  5. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  6. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  7. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 161
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  9. 2021Learning and Planning in Complex Action Spacescited by 77
  10. 2017Auto-Encoding Sequential Monte Carlocited by 65
  11. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  12. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
6 more
  1. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  2. 2022Critic Sequential Monte Carlocited by 16
  3. 2018Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Searchcited by 13
  4. 2022Epistemic Monte Carlo Tree Searchcited by 7
  5. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 5
  6. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Controlcited by 2
  2. 2026Particle Monte Carlo Tree Searchcited by 0
  3. 2026Towards Optimal Policy Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Variance-Aware Prior-Based Tree Policies for Monte Carlo Tree Search
  2. 2026Is Monte Carlo Tree Search Just Every-Visit Monte Carlo Control?
  3. 2024Optimized Monte Carlo Tree Search for Enhanced Decision Making in the FrozenLake Environment
  4. 2024Enhancing Reinforcement Learning Through Guided Search
  5. 2026Online Robust Reinforcement Learning Through Monte-Carlo Planning
  6. 2025TransZero: Parallel Tree Expansion in MuZero using Transformer Networks
  7. 2024Decision Making in Non-Stationary Environments with Policy-Augmented Search
  8. 2026Adaptive Parallel Monte Carlo Tree Search for Efficient Test-time Compute Scaling