# How to Evaluate Behavioral Models

arXiv: https://arxiv.org/abs/2306.04778 · HTML: https://mlsanity.com/paper/2306.04778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Predicting Human Behavior in Unrepeated, Simultaneous-Move Games](https://mlsanity.com/paper/1306.0918/) (2013)
- [Measuring the Completeness of Theories](https://mlsanity.com/paper/1910.07022/) (2019)
- [Neural Networks for Predicting Human Interactions in Repeated Games](https://mlsanity.com/paper/1911.03233/) (2019)
- [Toward a Characterization of Loss Functions for Distribution Learning](https://mlsanity.com/paper/1906.02652/) (2019)
- [Bridging Level-K to Nash Equilibrium](https://mlsanity.com/paper/2202.12292/) (2022)

## Abstract (arXiv)

Researchers building behavioral models, such as behavioral game theorists, use experimental data to evaluate predictive models of human behavior. However, there is little agreement about which loss function should be used in evaluations, with error rate, negative log-likelihood, cross-entropy, Brier score, and squared L2 error all being common choices. We attempt to offer a principled answer to the question of which loss functions should be used for this task, formalizing axioms that we argue loss functions should satisfy. We construct a family of loss functions, which we dub "diagonal bounded Bregman divergences", that satisfy all of these axioms. These rule out many loss functions used in practice, but notably include squared L2 error; we thus recommend its use for evaluating behavioral models.