# QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships

arXiv: https://arxiv.org/abs/1811.08048 · HTML: https://mlsanity.com/paper/1811.08048/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2018.

## Built on by (most cited first, 30 of 46)

- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://mlsanity.com/paper/2204.05660/) (2022)
- [Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2402.13950/) (2024)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements](https://mlsanity.com/paper/2305.03695/) (2023)
- [Pre-Training to Learn in Context](https://mlsanity.com/paper/2305.09137/) (2023)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners](https://mlsanity.com/paper/2210.02969/) (2022)
- [ILDAE: Instance-Level Difficulty Analysis of Evaluation Data](https://mlsanity.com/paper/2203.03073/) (2022)
- [Investigating Mysteries of CoT-Augmented Distillation](https://mlsanity.com/paper/2406.14511/) (2024)
- ["John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility](https://mlsanity.com/paper/2210.07471/) (2022)
- [Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization](https://mlsanity.com/paper/2210.09175/) (2022)
- [Crystal: Introspective Reasoners Reinforced with Self-Feedback](https://mlsanity.com/paper/2310.04921/) (2023)
- [Prompting with Pseudo-Code Instructions](https://mlsanity.com/paper/2305.11790/) (2023)
- [Lila: A Unified Benchmark for Mathematical Reasoning](https://mlsanity.com/paper/2210.17517/) (2022)
- [Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts](https://mlsanity.com/paper/2205.12701/) (2022)
- [Reliable Natural Language Understanding with Large Language Models and Answer Set Programming](https://mlsanity.com/paper/2302.03780/) (2023)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [Leaf: Multiple-Choice Question Generation](https://mlsanity.com/paper/2201.09012/) (2022)
- [Exploring Demonstration Ensembling for In-context Learning](https://mlsanity.com/paper/2308.08780/) (2023)

## Abstract (arXiv)

Many natural language questions require recognizing and reasoning with qualitative relationships (e.g., in science, economics, and medicine), but are challenging to answer with corpus-based methods. Qualitative modeling provides tools that support such reasoning, but the semantic parsing task of mapping questions into those models has formidable challenges. We present QuaRel, a dataset of diverse story questions involving qualitative relationships that characterize these challenges, and techniques that begin to address them. The dataset has 2771 questions relating 19 different types of quantities. For example, "Jenny observes that the robot vacuum cleaner moves slower on the living room carpet than on the bedroom carpet. Which carpet has more friction?" We contribute (1) a simple and flexible conceptual framework for representing these kinds of questions; (2) the QuaRel dataset, including logical forms, exemplifying the parsing challenges; and (3) two novel models for this task, built as extensions of type-constrained semantic parsing. The first of these models (called QuaSP+) significantly outperforms off-the-shelf tools on QuaRel. The second (QuaSP+Zero) demonstrates zero-shot capability, i.e., the ability to handle new qualitative relationships without requiring additional training data, something not possible with previous models. This work thus makes inroads into answering complex, qualitative questions that require reasoning, and scaling to new relationships at low cost. The dataset and models are available at http://data.allenai.org/quarel.