# FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information

arXiv: https://arxiv.org/abs/2305.01528 · HTML: https://mlsanity.com/paper/2305.01528/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 48 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence](https://mlsanity.com/paper/2210.07109/) (2022)
- [Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge](https://mlsanity.com/paper/1608.03000/) (2016)
- [Decoding Methods for Neural Narrative Generation](https://mlsanity.com/paper/2010.07375/) (2020)
- [Telling Stories through Multi-User Dialogue by Modeling Character Relations](https://mlsanity.com/paper/2105.15054/) (2021)

## Built on by (most cited first, 18 of 18)

- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [CALYPSO: LLMs as Dungeon Masters' Assistants](https://mlsanity.com/paper/2308.07540/) (2023)
- [Large Language Models and Video Games: A Preliminary Scoping Review](https://mlsanity.com/paper/2403.02613/) (2024)
- [PIPPA: A Partially Synthetic Conversational Dataset](https://mlsanity.com/paper/2308.05884/) (2023)
- [AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior](https://mlsanity.com/paper/2403.13433/) (2024)
- [Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models](https://mlsanity.com/paper/2307.12573/) (2023)
- [You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling](https://mlsanity.com/paper/2409.06949/) (2024)
- [Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-playing Games](https://mlsanity.com/paper/2309.13702/) (2023)
- [Reinforcement Learning Environment with LLM-Controlled Adversary in D&D 5th Edition Combat](https://mlsanity.com/paper/2503.15726/) (2025)
- [AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents](https://mlsanity.com/paper/2401.06509/) (2024)
- [Using Natural Language Inference to Improve Persona Extraction from Dialogue in a New Domain](https://mlsanity.com/paper/2401.06742/) (2024)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Large Language Model based Situational Dialogues for Second Language Learning](https://mlsanity.com/paper/2403.20005/) (2024)
- [Collaborative Storytelling and LLM: A Linguistic Analysis of Automatically-Generated Role-Playing Game Sessions](https://mlsanity.com/paper/2503.20623/) (2025)
- [Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG](https://mlsanity.com/paper/2607.02802/) (2026)
- [DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat](https://mlsanity.com/paper/2607.29577/) (2026)

## Abstract (arXiv)

Dungeons & Dragons (D&D) is a tabletop roleplaying game with complex natural language interactions between players and hidden state information. Recent work has shown that large language models (LLMs) that have access to state information can generate higher quality game turns than LLMs that use dialog history alone. However, previous work used game state information that was heuristically created and was not a true gold standard game state. We present FIREBALL, a large dataset containing nearly 25,000 unique sessions from real D&D gameplay on Discord with true game state info. We recorded game play sessions of players who used the Avrae bot, which was developed to aid people in playing D&D online, capturing language, game commands and underlying game state information. We demonstrate that FIREBALL can improve natural language generation (NLG) by using Avrae state information, improving both automated metrics and human judgments of quality. Additionally, we show that LLMs can generate executable Avrae commands, particularly after finetuning.