# Investigating Scale Independent UCT Exploration Factor Strategies

arXiv: https://arxiv.org/abs/2510.21275 · HTML: https://mlsanity.com/paper/2510.21275/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 24 Oct 2025.

## Builds on (most cited first, 5 of 5)

- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [The Design Of "Stratega": A General Strategy Games Framework](https://mlsanity.com/paper/2009.05643/) (2020)
- [The Second Type of Uncertainty in Monte Carlo Tree Search](https://mlsanity.com/paper/2005.09645/) (2020)
- [Evolving the MCTS Upper Confidence Bounds for Trees Using a Semantic-inspired Evolutionary Algorithm in the Game of Carcassonne](https://mlsanity.com/paper/2208.13589/) (2022)

## Built on by (most cited first, 4 of 4)

- [AUPO -- Abstracted Until Proven Otherwise: A Reward Distribution Based Abstraction Algorithm](https://mlsanity.com/paper/2510.23214/) (2025)
- [Investigating Intra-Abstraction Policies For Non-exact Abstraction Algorithms](https://mlsanity.com/paper/2510.24297/) (2025)
- [Grouping Nodes With Known Value Differences: A Lossless UCT-based Abstraction Algorithm](https://mlsanity.com/paper/2510.25388/) (2025)
- [Discovering State Equivalences in UCT Search Trees By Action Pruning](https://mlsanity.com/paper/2510.26346/) (2025)

## Abstract (arXiv)

The Upper Confidence Bounds For Trees (UCT) algorithm is not agnostic to the reward scale of the game it is applied to. For zero-sum games with the sparse rewards of -1,0,1 at the end of the game, this is not a problem, but many games often feature dense rewards with hand-picked reward scales, causing a node's Q-value to span different magnitudes across different games. In this paper, we evaluate various strategies for adaptively choosing the UCT exploration constant λ, called λ-strategies, that are agnostic to the game's reward scale. These λ-strategies include those proposed in the literature as well as five new strategies. Given our experimental results, we recommend using one of our newly suggested λ-strategies, which is to choose λ as 2 · σ where σ is the empirical standard deviation of all state-action pairs' Q-values of the search tree. This method outperforms existing λ-strategies across a wide range of tasks both in terms of a single parameter value and the peak performances obtained by optimizing all available parameters.