# Score vs. Winrate in Score-Based Games: which Reward for Reinforcement Learning?

arXiv: https://arxiv.org/abs/2201.13176 · HTML: https://mlsanity.com/paper/2201.13176/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 14 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 3 of 3)

- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [SAI, a Sensible Artificial Intelligence that plays Go](https://mlsanity.com/paper/1809.03928/) (2018)
- [SAI: a Sensible Artificial Intelligence that plays with handicap and targets high scores in 9x9 Go (extended version)](https://mlsanity.com/paper/1905.10863/) (2019)

## Built on by (most cited first, 2 of 2)

- [Increasing biases can be more efficient than increasing weights](https://mlsanity.com/paper/2301.00924/) (2023)
- [A Systematization of the Wagner Framework: Graph Theory Conjectures and Reinforcement Learning](https://mlsanity.com/paper/2406.12667/) (2024)

## Abstract (arXiv)

In the last years, the DeepMind algorithm AlphaZero has become the state of the art to efficiently tackle perfect information two-player zero-sum games with a win/lose outcome. However, when the win/lose outcome is decided by a final score difference, AlphaZero may play score-suboptimal moves because all winning final positions are equivalent from the win/lose outcome perspective. This can be an issue, for instance when used for teaching, or when trying to understand whether there is a better move. Moreover, there is the theoretical quest for the perfect game. A naive approach would be training an AlphaZero-like agent to predict score differences instead of win/lose outcomes. Since the game of Go is deterministic, this should as well produce an outcome-optimal play. However, it is a folklore belief that "this does not work". In this paper, we first provide empirical evidence for this belief. We then give a theoretical interpretation of this suboptimality in general perfect information two-player zero-sum game where the complexity of a game like Go is replaced by the randomness of the environment. We show that an outcome-optimal policy has a different preference for uncertainty when it is winning or losing. In particular, when in a losing state, an outcome-optimal agent chooses actions leading to a higher score variance. We then posit that when approximation is involved, a deterministic game behaves like a nondeterministic game, where the score variance is modeled by how uncertain the position is. We validate this hypothesis in AlphaZero-like software with a human expert.