# NARRA-Gym for Evaluating Interactive Narrative Agents

arXiv: https://arxiv.org/abs/2605.08503 · HTML: https://mlsanity.com/paper/2605.08503/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 47 of its 53 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 47)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://mlsanity.com/paper/2305.13304/) (2023)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [SEED-Story: Multimodal Long Story Generation with Large Language Model](https://mlsanity.com/paper/2407.08683/) (2024)
- [OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics](https://mlsanity.com/paper/2105.08920/) (2021)
- [UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation](https://mlsanity.com/paper/2009.07602/) (2020)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generation](https://mlsanity.com/paper/2407.01093/) (2024)
- [Learning to Reason for Long-Form Story Generation](https://mlsanity.com/paper/2503.22828/) (2025)
- [Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement](https://mlsanity.com/paper/2412.13575/) (2024)
- [Learning Personalized Alignment for Evaluating Open-ended Text Generation](https://mlsanity.com/paper/2310.03304/) (2023)
- [LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation](https://mlsanity.com/paper/2108.12960/) (2021)
- [CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints](https://mlsanity.com/paper/2410.04197/) (2024)
- [Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2212.04634/) (2022)
- [Creating Suspenseful Stories: Iterative Planning with Large Language Models](https://mlsanity.com/paper/2402.17119/) (2024)

## Built on by (most cited first, 1 of 1)

- [Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias](https://mlsanity.com/paper/2607.11871/) (2026)

## Abstract (arXiv)

Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality.