mlsanityData to 7 Oct 2026

Reasoning Over Paragraph Effects in Situations

arXiv 1908.05852 ↗First posted 16 Aug 2019cs.CLcited by 79

Not on the map. Why?

Abstract (arXiv)

A key component of successfully reading a passage of text is the ability to apply knowledge gained from the passage to a new situation. In order to facilitate progress on this kind of reading, we present ROPES, a challenging benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. We target expository language describing causes and effects (e.g., "animal pollinators increase efficiency of fertilization in flowers"), as they have clear implications for new situations. A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation. We collect background passages from science textbooks and Wikipedia that contain such phenomena, and ask crowd workers to author situations, questions, and answers, resulting in a 14,322 question dataset. We analyze the challenges of this task and evaluate the performance of state-of-the-art reading comprehension models. The best model performs only slightly better than randomly guessing an answer of the correct type, at 61.6% F1, well below the human performance of 89.0%.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 18; 2023: 20; 2024: 16; 2025: 4; 2026: 6.

79 papers.

Sort:

By area:

  1. 2022Galactica: A Large Language Model for Sciencecited by 736
  2. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  3. 2021MetaICL: Learning to Learn In Contextcited by 450
  4. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  5. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  8. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  9. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  10. 2022On the Paradox of Learning to Reason from Datacited by 93
  11. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
38 more
  1. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  2. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  3. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  4. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativenesscited by 76
  5. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  6. 2023Natural Language Reasoning, A Surveycited by 61
  7. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  8. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  9. 2024What Matters for Model Merging at Scale?cited by 52
  10. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  11. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  12. 2023Pre-Training to Learn in Contextcited by 41
  13. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  14. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  15. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  16. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  17. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
  18. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  19. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  20. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  21. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  22. 2021DREAM: Improving Situational QA by First Elaborating the Situationcited by 18
  23. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logiccited by 18
  24. 2022RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Questioncited by 17
  25. 2021multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoningcited by 15
  26. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  27. 2022BioTABQA: Instruction Learning for Biomedical Table Question Answeringcited by 13
  28. 2023Prompting with Pseudo-Code Instructionscited by 13
  29. 2021On the Influence of Masking Policies in Intermediate Pre-trainingcited by 12
  30. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  31. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11
  32. 2022Interpretable Proof Generation via Iterative Backward Reasoningcited by 9
  33. 2021Learning with Instance Bundles for Reading Comprehensioncited by 8
  34. 2023Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answeringcited by 8
  35. 2023Text Alignment Is An Efficient Unified Model for Massive NLP Taskscited by 8
  36. 2023Long-Tailed Question Answering in an Open Worldcited by 7
  37. 2023Teaching Probabilistic Logical Reasoning to Transformerscited by 7
  38. 2024Paraphrasing in Affirmative Terms Improves Negation Understandingcited by 7

50 most cited of 79 All 79 (JSON)