mlsanityData to 7 Oct 2026

EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

arXiv 1901.03735 ↗First posted 11 Jan 2019cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle. We present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quantities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2%), but has limited verbal reasoning capabilities (-8.1%). We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 14; 2023: 7; 2024: 6; 2025: 1; 2026: 1.

39 papers.

Sort:

By area:

  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  3. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  4. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  5. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  6. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  7. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  8. 2022Reasoning Like Program Executorscited by 45
  9. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  10. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  11. 2021IndoNLI: A Natural Language Inference Dataset for Indonesiancited by 24
  12. 2024QuanTemp: A real-world open-domain benchmark for fact-checking numerical claimscited by 22
27 more
  1. 2024SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trialscited by 20
  2. 2023NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reportscited by 18
  3. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  4. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  5. 2022Chaining Simultaneous Thoughts for Numerical Reasoningcited by 14
  6. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  7. 2022Lila: A Unified Benchmark for Mathematical Reasoningcited by 13
  8. 2021NT5?! Training T5 to Perform Numerical Reasoningcited by 11
  9. 2021Numerical reasoning in machine reading comprehension tasks: are we there yet?cited by 9
  10. 2022A Neural-Symbolic Approach to Natural Language Understandingcited by 8
  11. 2022Let the Model Decide its Curriculum for Multitask Learningcited by 8
  12. 2022EvEntS ReaLM: Event Reasoning of Entity States via Language Modelscited by 7
  13. 2023SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Datacited by 6
  14. 2021Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Contextcited by 5
  15. 2023GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmarkcited by 5
  16. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  17. 2022Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmeticcited by 4
  18. 2023Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?cited by 4
  19. 2023Enhancing Quantitative Reasoning Skills of Large Language Models through Dimension Perceptioncited by 4
  20. 2023Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Modelscited by 2
  21. 2024From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisitioncited by 2
  22. 2022nBIIG: A Neural BI Insights Generation System for Table Reportingcited by 1
  23. 2025Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?cited by 1
  24. 2022Logical Inference for Counting on Semi-structured Tablescited by 0
  25. 2023Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inferencecited by 0
  26. 2024Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Modelscited by 0
  27. 2026E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understandingcited by 0