mlsanityData to 7 Oct 2026

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

arXiv 1909.03553 ↗First posted 8 Sep 2019cs.CL cs.AIcited by 89

Not on the map. Why?

Abstract (arXiv)

We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., "A sunscreen with a higher SPF protects the skin longer.", twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?", plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system's ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 23; 2023: 19; 2024: 14; 2025: 17; 2026: 4.

89 papers.

Sort:

By area:

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2024Scaling and evaluating sparse autoencoderscited by 530
  4. 2021MetaICL: Learning to Learn In Contextcited by 450
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  7. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  8. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  9. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  10. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  11. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  12. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
38 more
  1. 2024Parameter Competition Balancing for Model Mergingcited by 71
  2. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
  3. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  4. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  5. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  6. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  7. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  8. 2023Pre-Training to Learn in Contextcited by 41
  9. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  10. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  11. 2022Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvementcited by 37
  12. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  13. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  14. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
  15. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  16. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  17. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  18. 2022REV: Information-Theoretic Evaluation of Free-Text Rationalescited by 22
  19. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  20. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logiccited by 18
  21. 2022ILDAE: Instance-Level Difficulty Analysis of Evaluation Datacited by 17
  22. 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
  23. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  24. 2021multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoningcited by 15
  25. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  26. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  27. 2023Language Models with Rationalitycited by 15
  28. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15
  29. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  30. 2022Investigating the Benefits of Free-Form Rationalescited by 13
  31. 2025Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generationcited by 13
  32. 2021On the Influence of Masking Policies in Intermediate Pre-trainingcited by 12
  33. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  34. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
  35. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  36. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11
  37. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11
  38. 2025Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfercited by 11

50 most cited of 89 All 89 (JSON)