# Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension

arXiv: https://arxiv.org/abs/2203.13947 · HTML: https://mlsanity.com/paper/2203.13947/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- ["Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding](https://mlsanity.com/paper/2109.05438/) (2021)
- [StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement](https://mlsanity.com/paper/2202.06205/) (2022)
- [Exploring Content Selection in Summarization of Novel Chapters](https://mlsanity.com/paper/2005.01840/) (2020)
- [Large-scale Cloze Test Dataset Created by Teachers](https://mlsanity.com/paper/1711.03225/) (2017)
- [TellMeWhy: A Dataset for Answering Why-Questions in Narratives](https://mlsanity.com/paper/2106.06132/) (2021)
- [It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books](https://mlsanity.com/paper/2109.03423/) (2021)
- [Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study](https://mlsanity.com/paper/2106.03826/) (2021)
- [Embracing data abundance: BookTest Dataset for Reading Comprehension](https://mlsanity.com/paper/1610.00956/) (2016)

## Built on by (most cited first, 30 of 80)

- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Learning to Filter Context for Retrieval-Augmented Generation](https://mlsanity.com/paper/2311.08377/) (2023)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [GEMv2: Multilingual NLG Benchmarking in a Single Line of Code](https://mlsanity.com/paper/2206.11249/) (2022)
- [Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models](https://mlsanity.com/paper/2412.14528/) (2024)
- [Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation](https://mlsanity.com/paper/2507.21028/) (2025)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs](https://mlsanity.com/paper/2408.08656/) (2024)
- [Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild](https://mlsanity.com/paper/2505.16023/) (2025)
- [PaniniQA: Enhancing Patient Education Through Interactive Question Answering](https://mlsanity.com/paper/2308.03253/) (2023)
- [Personality Understanding of Fictional Characters during Book Reading](https://mlsanity.com/paper/2305.10156/) (2023)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [TVShowGuess: Character Comprehension in Stories as Speaker Guessing](https://mlsanity.com/paper/2204.07721/) (2022)
- [Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation](https://mlsanity.com/paper/2209.11000/) (2022)
- [MBTI Personality Prediction for Fictional Characters Using Movie Scripts](https://mlsanity.com/paper/2210.10994/) (2022)
- [Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives](https://mlsanity.com/paper/2402.11051/) (2024)
- [Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations](https://mlsanity.com/paper/2305.03117/) (2023)
- [Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank](https://mlsanity.com/paper/2306.08847/) (2023)
- [Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective](https://mlsanity.com/paper/2410.17600/) (2024)
- [Discord Questions: A Computational Approach To Diversity Analysis in News Coverage](https://mlsanity.com/paper/2211.05007/) (2022)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions](https://mlsanity.com/paper/2402.13551/) (2024)
- [A Survey of Machine Narrative Reading Comprehension Assessments](https://mlsanity.com/paper/2205.00299/) (2022)
- [Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture](https://mlsanity.com/paper/2305.12710/) (2023)
- [NarrativePlay: Interactive Narrative Understanding](https://mlsanity.com/paper/2310.01459/) (2023)
- [QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation](https://mlsanity.com/paper/2210.04320/) (2022)

## Abstract (arXiv)

Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models' fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.