mlsanityData to 7 Oct 2026

TellMeWhy: A Dataset for Answering Why-Questions in Narratives

arXiv 2106.06132 ↗First posted 11 Jun 2021cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 21 of its 58 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

21 of 58 references.

References · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  9. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  10. 2019Abductive Commonsense Reasoningcited by 300
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
9 more
  1. 2020GLUCOSE: GeneraLized and COntextualized Story Explanationscited by 85
  2. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  3. 2020To Test Machine Comprehension, Start by Defining Comprehensioncited by 31
  4. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  5. 2018Efficient Online Scalar Annotation with Bounded Supportcited by 27
  6. 2020Author's Sentiment Predictioncited by 18
  7. 2019Towards Best Experiment Design for Evaluating Dialogue System Outputcited by 15
  8. 2020TuringAdvice: A Generative and Dynamic Evaluation of Language Usecited by 4
  9. 2019Learning to Explain: Answering Why-Questions via Rephrasingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 8; 2024: 8; 2025: 0; 2026: 3.

27 papers.

Sort:

By area:

  1. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 102
  2. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 79
  3. 2023Natural Language Reasoning, A Surveycited by 61
  4. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  5. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  6. 2022TVShowGuess: Character Comprehension in Stories as Speaker Guessingcited by 17
  7. 2022ALERT: Adapting Language Models to Reasoning Taskscited by 17
  8. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Modelscited by 13
  9. 2023Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rankcited by 12
  10. 2024ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Modelscited by 12
  11. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-trainingcited by 11
  12. 2024Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questionscited by 11
15 more
  1. 2022A Survey of Machine Narrative Reading Comprehension Assessmentscited by 10
  2. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Planscited by 10
  3. 2022Inferring Implicit Relations in Complex Questions with Language Modelscited by 9
  4. 2023NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillationcited by 8
  5. 2023Diversity Enhanced Narrative Question Generation for Storybookscited by 5
  6. 2023Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customizationcited by 5
  7. 2022rmC small IS2: A Simplified Commonsense Inference Evaluation for Story Prosecited by 4
  8. 2023Automatic Evaluation of Generative Models with Instruction Tuningcited by 4
  9. 2024SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Eventscited by 2
  10. 2024Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledgecited by 1
  11. 2026Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMscited by 1
  12. 2024WHAT-IF: Exploring Branching Narratives by Meta-Prompting Large Language Modelscited by 0
  13. 2024DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answeringcited by 0
  14. 2026Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Informationcited by 0
  15. 2026Contextual Causality with Large Language Models: A Surveycited by 0