# TellMeWhy: A Dataset for Answering Why-Questions in Narratives

arXiv: https://arxiv.org/abs/2106.06132 · HTML: https://mlsanity.com/paper/2106.06132/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 58 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [GLUCOSE: GeneraLized and COntextualized Story Explanations](https://mlsanity.com/paper/2009.07758/) (2020)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [To Test Machine Comprehension, Start by Defining Comprehension](https://mlsanity.com/paper/2005.01525/) (2020)
- [Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale](https://mlsanity.com/paper/2010.13588/) (2020)
- [Efficient Online Scalar Annotation with Bounded Support](https://mlsanity.com/paper/1806.01170/) (2018)
- [Author's Sentiment Prediction](https://mlsanity.com/paper/2011.06128/) (2020)
- [Towards Best Experiment Design for Evaluating Dialogue System Output](https://mlsanity.com/paper/1909.10122/) (2019)
- [TuringAdvice: A Generative and Dynamic Evaluation of Language Use](https://mlsanity.com/paper/2004.03607/) (2020)
- [Learning to Explain: Answering Why-Questions via Rephrasing](https://mlsanity.com/paper/1906.01243/) (2019)

## Built on by (most cited first, 27 of 27)

- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [WikiWhy: Answering and Explaining Cause-and-Effect Questions](https://mlsanity.com/paper/2210.12152/) (2022)
- [TVShowGuess: Character Comprehension in Stories as Speaker Guessing](https://mlsanity.com/paper/2204.07721/) (2022)
- [ALERT: Adapting Language Models to Reasoning Tasks](https://mlsanity.com/paper/2212.08286/) (2022)
- [A Critical Review of Causal Reasoning Benchmarks for Large Language Models](https://mlsanity.com/paper/2407.08029/) (2024)
- [Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank](https://mlsanity.com/paper/2306.08847/) (2023)
- [ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Models](https://mlsanity.com/paper/2402.00794/) (2024)
- [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://mlsanity.com/paper/2205.08794/) (2022)
- [Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions](https://mlsanity.com/paper/2402.13551/) (2024)
- [CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans](https://mlsanity.com/paper/2406.15823/) (2024)
- [A Survey of Machine Narrative Reading Comprehension Assessments](https://mlsanity.com/paper/2205.00299/) (2022)
- [Inferring Implicit Relations in Complex Questions with Language Models](https://mlsanity.com/paper/2204.13778/) (2022)
- [NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation](https://mlsanity.com/paper/2312.05979/) (2023)
- [Diversity Enhanced Narrative Question Generation for Storybooks](https://mlsanity.com/paper/2310.16446/) (2023)
- [Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization](https://mlsanity.com/paper/2311.09510/) (2023)
- [rmC small IS2: A Simplified Commonsense Inference Evaluation for Story Prose](https://mlsanity.com/paper/2202.07880/) (2022)
- [Automatic Evaluation of Generative Models with Instruction Tuning](https://mlsanity.com/paper/2310.20072/) (2023)
- [SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events](https://mlsanity.com/paper/2408.05793/) (2024)
- [Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs](https://mlsanity.com/paper/2604.13258/) (2026)
- [Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledge](https://mlsanity.com/paper/2408.01453/) (2024)
- [WHAT-IF: Exploring Branching Narratives by Meta-Prompting Large Language Models](https://mlsanity.com/paper/2412.10582/) (2024)
- [DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answering](https://mlsanity.com/paper/2412.16694/) (2024)
- [Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information](https://mlsanity.com/paper/2601.03089/) (2026)
- [Contextual Causality with Large Language Models: A Survey](https://mlsanity.com/paper/2609.22409/) (2026)

## Abstract (arXiv)

Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.