# Assessing AI Utility: The Random Guesser Test for Sequential Decision-Making Systems

arXiv: https://arxiv.org/abs/2407.20276 · HTML: https://mlsanity.com/paper/2407.20276/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 23 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 4 of 4)

- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [The Use of Machine Learning Algorithms in Recommender Systems: A Systematic Review](https://mlsanity.com/paper/1511.05263/) (2015)
- [A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry](https://mlsanity.com/paper/1906.11286/) (2019)
- [Bandit Models of Human Behavior: Reward Processing in Mental Disorders](https://mlsanity.com/paper/1706.02897/) (2017)

## Built on by (most cited first, 1 of 1)

- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)

## Abstract (arXiv)

We propose a general approach to quantitatively assessing the risk and vulnerability of artificial intelligence (AI) systems to biased decisions. The guiding principle of the proposed approach is that any AI algorithm must outperform a random guesser. This may appear trivial, but empirical results from a simplistic sequential decision-making scenario involving roulette games show that sophisticated AI-based approaches often underperform the random guesser by a significant margin. We highlight that modern recommender systems may exhibit a similar tendency to favor overly low-risk options. We argue that this "random guesser test" can serve as a useful tool for evaluating the utility of AI actions, and also points towards increasing exploration as a potential improvement to such systems.