mlsanityData to 7 Oct 2026

Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios

arXiv 2209.07760 ↗First posted 16 Sep 2022cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The possible consequences for the same context may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). Through a comparison with existing datasets, we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that require counterfactual reasoning, as well as those requiring readers' reactions and fictional information, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 61 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

39 of 61 references.

References · 39

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
27 more
  1. 2019Abductive Commonsense Reasoningcited by 300
  2. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  3. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  4. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  6. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  7. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  8. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  9. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  10. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  11. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  12. 2019Counterfactual Story Reasoning and Generationcited by 87
  13. 2020GLUCOSE: GeneraLized and COntextualized Story Explanationscited by 85
  14. 2019WIQA: A dataset for "What if..." reasoning over procedural textcited by 71
  15. 2018Modeling Naive Psychology of Characters in Simple Commonsense Storiescited by 59
  16. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  17. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  18. 2019Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasetscited by 41
  19. 2016Ordinal Common-sense Inferencecited by 35
  20. 2019Uncertain Natural Language Inferencecited by 35
  21. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 30
  22. 2021What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?cited by 23
  23. 2021Embracing Ambiguity: Shifting the Training Target of NLI Modelscited by 22
  24. 2021Modeling Event Plausibility with Consistent Conceptual Abstractioncited by 20
  25. 2021Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledgecited by 15
  26. 2021Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcingcited by 11
  27. 2022What Makes Reading Comprehension Questions Difficult?cited by 11

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  2. 2023A Study of Situational Reasoning for Traffic Understandingcited by 13
  3. 2024Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022rmC small IS2: A Simplified Commonsense Inference Evaluation for Story Prose
  2. 2023SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning
  3. 2023An Overview Of Temporal Commonsense Reasoning and Acquisition
  4. 2024Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
  5. 2024Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?
  6. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
  7. 2023Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
  8. 2024What Really is Commonsense Knowledge?