mlsanityData to 14 Oct 2026

Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control

arXiv 2505.09029 ↗First posted 13 May 2025cs.AI cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), depend on basic noise-based exploration, which can result in less than optimal policy convergence. In this study, we introduce Monte Carlo Beam Search (MCBS), a new hybrid method that combines beam search and Monte Carlo rollouts with TD3 to improve exploration and action selection. MCBS produces several candidate actions around the policy's output and assesses them through short-horizon rollouts, enabling the agent to make better-informed choices. We test MCBS across various continuous-control benchmarks, including HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, showing enhanced sample efficiency and performance compared to standard TD3 and other baseline methods like SAC, PPO, and A2C. Our findings emphasize MCBS's capability to enhance policy learning through structured look-ahead search while ensuring computational efficiency. Additionally, we offer a detailed analysis of crucial hyperparameters, such as beam width and rollout depth, and explore adaptive strategies to optimize MCBS for complex control tasks. Our method shows a higher convergence rate across different environments compared to TD3, SAC, PPO, and A2C. For instance, we achieved 90% of the maximum achievable reward within around 200 thousand timesteps compared to 400 thousand timesteps for the second-best method.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 23 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

15 of 23 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2016OpenAI Gymcited by 1,736
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  6. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  7. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 161
  8. 2022Simulation-guided Beam Search for Neural Combinatorial Optimizationcited by 84
  9. 2025MuJoCo Playgroundcited by 78
  10. 2024A comprehensive survey of research towards AI-enabled unmanned aerial systems in pre-, active-, and post-wildfire managementcited by 25
  11. 2022A2C is a special case of PPOcited by 16
  12. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 14
3 more
  1. 2022Using Reinforcement Learning for Load Testing of Video Gamescited by 11
  2. 2022Beyond Greedy Search: Tracking by Multi-Agent Reinforcement Learning-based Beam Searchcited by 6
  3. 2023Actuator Trajectory Planning for UAVs with Overhead Manipulator using Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 7
  2. 2026Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Modelscited by 1
  3. 2026Meta-Adaptive Beam Search Planning for Transformer-Based Reinforcement Learning Control of UAVs with Overhead Manipulators under Flight Disturbancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
  2. 2024SPO: Sequential Monte Carlo Policy Optimisation
  3. 2025Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization
  4. 2025Variance-Aware Prior-Based Tree Policies for Monte Carlo Tree Search
  5. 2026Bayesian policy gradient and actor-critic algorithms
  6. 2024Optimized Monte Carlo Tree Search for Enhanced Decision Making in the FrozenLake Environment
  7. 2026Particle Monte Carlo Tree Search
  8. 2024Decision Making in Non-Stationary Environments with Policy-Augmented Search