# Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-playing Games

arXiv: https://arxiv.org/abs/2309.13702 · HTML: https://mlsanity.com/paper/2309.13702/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 24 Sep 2023.

## Builds on (most cited first, 11 of 11)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Narrative Planning: Balancing Plot and Character](https://mlsanity.com/paper/1401.3841/) (2014)
- [Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence](https://mlsanity.com/paper/2210.07109/) (2022)
- [CALYPSO: LLMs as Dungeon Masters' Assistants](https://mlsanity.com/paper/2308.07540/) (2023)
- [Designing Participatory AI: Creative Professionals' Worries and Expectations about Generative AI](https://mlsanity.com/paper/2303.08931/) (2023)
- [FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information](https://mlsanity.com/paper/2305.01528/) (2023)
- [Machine Common Sense Concept Paper](https://mlsanity.com/paper/1810.07528/) (2018)
- [A guided journey through non-interactive automatic story generation](https://mlsanity.com/paper/2110.11167/) (2021)

## Built on by (most cited first, 3 of 3)

- [PAYADOR: A Minimalist Approach to Grounding Language Models on Structured Data for Interactive Storytelling and Role-playing Games](https://mlsanity.com/paper/2504.07304/) (2025)
- [World-State Transformations for Neuro-symbolic Interactive Storytelling](https://mlsanity.com/paper/2605.24719/) (2026)
- [IVIE: A Neuro-symbolic Approach to Incremental and Validated Generation of Interactive Fiction Worlds](https://mlsanity.com/paper/2606.13348/) (2026)

## Abstract (arXiv)

In role-playing games a Game Master (GM) is the player in charge of the game, who must design the challenges the players face and narrate the outcomes of their actions. In this work we discuss some challenges to model GMs from an Interactive Storytelling and Natural Language Processing perspective. Following those challenges we propose three test categories to evaluate such dialogue systems, and we use them to test ChatGPT, Bard and OpenAssistant as out-of-the-box GMs.