# FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs

arXiv: https://arxiv.org/abs/2508.17825 · HTML: https://mlsanity.com/paper/2508.17825/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement](https://mlsanity.com/paper/2402.11436/) (2024)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation](https://mlsanity.com/paper/2309.17234/) (2023)
- [Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis](https://mlsanity.com/paper/2312.05488/) (2023)
- [ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark](https://mlsanity.com/paper/2303.13648/) (2023)
- [MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration](https://mlsanity.com/paper/2311.08562/) (2023)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory](https://mlsanity.com/paper/2408.02784/) (2024)
- [GameGPT: Multi-agent Collaborative Framework for Game Development](https://mlsanity.com/paper/2310.08067/) (2023)
- [Fake Alignment: Are LLMs Really Aligned Well?](https://mlsanity.com/paper/2311.05915/) (2023)
- [GameEval: Evaluating LLMs on Conversational Games](https://mlsanity.com/paper/2308.10032/) (2023)
- [Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models](https://mlsanity.com/paper/2408.03907/) (2024)
- [FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity](https://mlsanity.com/paper/2311.18580/) (2023)
- [Player-Driven Emergence in LLM-Driven Game Narrative](https://mlsanity.com/paper/2404.17027/) (2024)
- [AI Sees Your Location, But With A Bias Toward The Wealthy World](https://mlsanity.com/paper/2502.11163/) (2025)
- [Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases](https://mlsanity.com/paper/2502.05849/) (2025)
- [Game Plot Design with an LLM-powered Assistant: An Empirical Study with Game Designers](https://mlsanity.com/paper/2411.02714/) (2024)
- [General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level](https://mlsanity.com/paper/2311.13892/) (2023)

## Built on by (most cited first, 1 of 1)

- [RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents](https://mlsanity.com/paper/2604.11655/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have increasingly enhanced or replaced traditional Non-Player Characters (NPCs) in video games. However, these LLM-based NPCs inherit underlying social biases (e.g., race or class), posing fairness risks during in-game interactions. To address the limited exploration of this issue, we introduce FairGamer, the first benchmark to evaluate social biases across three interaction patterns: transaction, cooperation, and competition. FairGamer assesses four bias types, including class, race, age, and nationality, across 12 distinct evaluation tasks using a novel metric, FairMCV. Our evaluation of seven frontier LLMs reveals that: (1) models exhibit biased decision-making, with Grok-4-Fast demonstrating the highest bias (average FairMCV = 76.9%); and (2) larger LLMs display more severe social biases, suggesting that increased model capacity inadvertently amplifies these biases. We release FairGamer at https://github.com/Anonymous999-xxx/FairGamer to facilitate future research on NPC fairness.