# The Evaluation of Rating Systems in Team-based Battle Royale Games

arXiv: https://arxiv.org/abs/2105.14069 · HTML: https://mlsanity.com/paper/2105.14069/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 26 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 4 of 4)

- [Competitive Balance in Team Sports Games](https://mlsanity.com/paper/2006.13763/) (2020)
- [The Evaluation of Rating Systems in Online Free-for-All Games](https://mlsanity.com/paper/2008.06787/) (2020)
- [Rapid Skill Capture in a First-Person Shooter](https://mlsanity.com/paper/1411.1316/) (2014)
- [MOBA-Slice: A Time Slice Based Evaluation Framework of Relative Advantage between Teams in MOBA Games](https://mlsanity.com/paper/1807.08360/) (2018)

## Built on by (most cited first, 3 of 3)

- [Evaluating Team Skill Aggregation in Online Competitive Games](https://mlsanity.com/paper/2106.11397/) (2021)
- [Player Modeling using Behavioral Signals in Competitive Online Games](https://mlsanity.com/paper/2112.04379/) (2021)
- [Behavioral Player Rating in Competitive Online Shooter Games](https://mlsanity.com/paper/2207.00528/) (2022)

## Abstract (arXiv)

Online competitive games have become a mainstream entertainment platform. To create a fair and exciting experience, these games use rating systems to match players with similar skills. While there has been an increasing amount of research on improving the performance of these systems, less attention has been paid to how their performance is evaluated. In this paper, we explore the utility of several metrics for evaluating three popular rating systems on a real-world dataset of over 25,000 team battle royale matches. Our results suggest considerable differences in their evaluation patterns. Some metrics were highly impacted by the inclusion of new players. Many could not capture the real differences between certain groups of players. Among all metrics studied, normalized discounted cumulative gain (NDCG) demonstrated more reliable performance and more flexibility. It alleviated most of the challenges faced by the other metrics while adding the freedom to adjust the focus of the evaluations on different groups of players.