# Reasoning Over Paragraph Effects in Situations

arXiv: https://arxiv.org/abs/1908.05852 · HTML: https://mlsanity.com/paper/1908.05852/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2019.

## Built on by (most cited first, 30 of 79)

- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [On the Paradox of Learning to Reason from Data](https://mlsanity.com/paper/2205.11502/) (2022)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [ChatQA: Surpassing GPT-4 on Conversational QA and RAG](https://mlsanity.com/paper/2401.10225/) (2024)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://mlsanity.com/paper/2304.10703/) (2023)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Memory-efficient Transformers via Top-k Attention](https://mlsanity.com/paper/2106.06899/) (2021)
- [UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training](https://mlsanity.com/paper/2202.12359/) (2022)
- [What Matters for Model Merging at Scale?](https://mlsanity.com/paper/2410.03617/) (2024)
- [Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions](https://mlsanity.com/paper/2205.00415/) (2022)
- [LongForm: Effective Instruction Tuning with Reverse Instructions](https://mlsanity.com/paper/2304.08460/) (2023)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Pre-Training to Learn in Context](https://mlsanity.com/paper/2305.09137/) (2023)
- [LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging](https://mlsanity.com/paper/2410.17146/) (2024)
- [Merging by Matching Models in Task Parameter Subspaces](https://mlsanity.com/paper/2312.04339/) (2023)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [FaiRR: Faithful and Robust Deductive Reasoning over Natural Language](https://mlsanity.com/paper/2203.10261/) (2022)

## Abstract (arXiv)

A key component of successfully reading a passage of text is the ability to apply knowledge gained from the passage to a new situation. In order to facilitate progress on this kind of reading, we present ROPES, a challenging benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. We target expository language describing causes and effects (e.g., "animal pollinators increase efficiency of fertilization in flowers"), as they have clear implications for new situations. A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation. We collect background passages from science textbooks and Wikipedia that contain such phenomena, and ask crowd workers to author situations, questions, and answers, resulting in a 14,322 question dataset. We analyze the challenges of this task and evaluate the performance of state-of-the-art reading comprehension models. The best model performs only slightly better than randomly guessing an answer of the correct type, at 61.6% F1, well below the human performance of 89.0%.