# Evolving choice hysteresis in reinforcement learning: comparing the adaptive value of positivity bias and gradual perseveration

arXiv: https://arxiv.org/abs/2410.19434 · HTML: https://mlsanity.com/paper/2410.19434/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 53 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 4 of 4)

- [A Survey on Practical Applications of Multi-Armed and Contextual Bandits](https://mlsanity.com/paper/1904.10040/) (2019)
- [In-context learning agents are asymmetric belief updaters](https://mlsanity.com/paper/2402.03969/) (2024)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [Improving Neural Network Learning Through Dual Variable Learning Rates](https://mlsanity.com/paper/2002.03428/) (2020)

## Built on by (most cited first, 1 of 1)

- [Weber-Fechner Law in Temporal Difference learning derived from Control as Inference](https://mlsanity.com/paper/2412.21004/) (2024)

## Abstract (arXiv)

The tendency of repeating past choices more often than expected from the history of outcomes has been repeatedly empirically observed in reinforcement learning experiments. It can be explained by at least two computational processes: asymmetric update and (gradual) choice perseveration. A recent meta-analysis showed that both mechanisms are detectable in human reinforcement learning. However, while their descriptive value seems to be well established, they have not been compared regarding their possible adaptive value. In this study, we address this gap by simulating reinforcement learning agents in a variety of environments with a new variant of an evolutionary algorithm. Our results show that positivity bias (in the form of asymmetric update) is evolutionary stable in many situations, while the emergence of gradual perseveration is less systematic and robust. Overall, our results illustrate that biases can be adaptive and selected by evolution, in an environment-specific manner.