mlsanityData to 7 Oct 2026

SPO: Sequential Monte Carlo Policy Optimisation

arXiv 2402.07963 ↗First posted 12 Feb 2024cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Leveraging planning during learning and decision-making is central to the long-term development of intelligent agents. Recent works have successfully combined tree-based search methods and self-play learning mechanisms to this end. However, these methods typically face scaling challenges due to the sequential nature of their search. While practical engineering solutions can partly overcome this, they often result in a negative impact on performance. In this paper, we introduce SPO: Sequential Monte Carlo Policy Optimisation, a model-based reinforcement learning algorithm grounded within the Expectation Maximisation (EM) framework. We show that SPO provides robust policy improvement and efficient scaling properties. The sample-based search makes it directly applicable to both discrete and continuous action spaces without modifications. We demonstrate statistically significant improvements in performance relative to model-free and model-based baselines across both continuous and discrete environments. Furthermore, the parallel nature of SPO's search enables effective utilisation of hardware accelerators, yielding favourable scaling laws.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 39 of its 85 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

39 of 85 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2018Attention, Learn to Solve Routing Problems!cited by 646
27 more
  1. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  3. 2018Maximum a Posteriori Policy Optimisationcited by 318
  4. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  5. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  7. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  8. 2022Tutorial on amortized optimizationcited by 83
  9. 2021Learning and Planning in Complex Action Spacescited by 76
  10. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  11. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  12. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  13. 2020Munchausen Reinforcement Learningcited by 68
  14. 2019An investigation of model-free planningcited by 64
  15. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 59
  16. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 52
  17. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  18. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  19. 2015Neural Adaptive Sequential Monte Carlocited by 34
  20. 2017Approximate Inference with Amortised MCMCcited by 22
  21. 2018Dual Policy Iterationcited by 22
  22. 2018A0C: Alpha Zero in Continuous Action Spacecited by 17
  23. 2022Critic Sequential Monte Carlocited by 16
  24. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  25. 2016Guided Policy Search as Approximate Mirror Descentcited by 14
  26. 2018Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Searchcited by 13
  27. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Trust-Region Twisted Policy Improvementcited by 5
  2. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  3. 2025Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategiescited by 2
  4. 2026VariBASed: Variational Bayes-Adaptive Sequential Monte-Carlo Planning for Deep Reinforcement Learningcited by 0