mlsanityData to 7 Oct 2026

LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios

arXiv 2310.08348 ↗First posted 12 Oct 2023cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Building agents based on tree-search planning capabilities with learned models has achieved remarkable success in classic decision-making problems, such as Go and Atari. However, it has been deemed challenging or even infeasible to extend Monte Carlo Tree Search (MCTS) based algorithms to diverse real-world applications, especially when these environments involve complex action spaces and significant simulation costs, or inherent stochasticity. In this work, we introduce LightZero, the first unified benchmark for deploying MCTS/MuZero in general sequential decision scenarios. Specificially, we summarize the most critical challenges in designing a general MCTS-style decision-making solver, then decompose the tightly-coupled algorithm and system design of tree-search RL methods into distinct sub-modules. By incorporating more appropriate exploration and optimization strategies, we can significantly enhance these sub-modules and construct powerful LightZero agents to tackle tasks across a wide range of domains, such as board games, Atari, MuJoCo, MiniGrid and GoBigger. Detailed benchmark results reveal the significant potential of such methods in building scalable and efficient decision intelligence. The code is available as part of OpenDILab at https://github.com/opendilab/LightZero.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 56 of its 80 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

56 of 80 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2016OpenAI Gymcited by 1,684
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2016An overview of gradient descent optimization algorithmscited by 1,021
  12. 2020Mastering Atari with Discrete World Modelscited by 990
38 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  13. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  14. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  15. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  16. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  17. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  18. 2018Distributed Prioritized Experience Replaycited by 260
  19. 2021Mastering Atari Games with Limited Datacited by 235
  20. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  21. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  22. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  23. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  24. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  25. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  26. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  27. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  28. 2019Behaviour Suite for Reinforcement Learningcited by 129
  29. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  30. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  31. 2016Learning values across many orders of magnitudecited by 80
  32. 2021Learning and Planning in Complex Action Spacescited by 76
  33. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  34. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  35. 2018Probabilistic Recurrent State-Space Modelscited by 64
  36. 2019Accelerating Self-Play Learning in Gocited by 60
  37. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 49
  38. 2021Vector Quantized Models for Planningcited by 47

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 5; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  2. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  3. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  4. 2024Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvementcited by 32
  5. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  6. 2023MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Gamescited by 14
  7. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  8. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  9. 2026DriveMamba: Task-Centric Scalable State Space Model for Efficient End-to-End Autonomous Drivingcited by 9
  10. 2024LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in Large Language Modelscited by 7
  11. 2024EgoFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Drivingcited by 3
  12. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
8 more
  1. 2024ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyzecited by 2
  2. 2024Demystifying MuZero Planning: Interpreting the Learned Modelcited by 1
  3. 2025Empowering LLMs in Decision Games through Algorithmic Data Synthesiscited by 1
  4. 2025Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learningcited by 1
  5. 2026PriorZero: Bridging Language Priors and World Models for Decision Makingcited by 1
  6. 2025Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learningcited by 0
  7. 2026Coupled Local and Global World Models for Efficient First Order RLcited by 0
  8. 2026Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversationscited by 0