mlsanityData to 14 Oct 2026

Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems

arXiv 2211.07455 ↗First posted 14 Nov 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Numerical Question Answering is the task of answering questions that require numerical capabilities. Previous works introduce general adversarial attacks to Numerical Question Answering, while not systematically exploring numerical capabilities specific to the topic. In this paper, we propose to conduct numerical capability diagnosis on a series of Numerical Question Answering systems and datasets. A series of numerical capabilities are highlighted, and corresponding dataset perturbations are designed. Empirical results indicate that existing systems are severely challenged by these perturbations. E.g., Graph2Tree experienced a 53.83% absolute accuracy drop against the ``Extra'' perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the ``Language'' perturbation on the numerical subset of DROP. As a counteracting approach, we also investigate the effectiveness of applying perturbations as data augmentation to relieve systems' lack of robust numerical capabilities. With experiment analysis and empirical studies, it is demonstrated that Numerical Question Answering with robust numerical capabilities is still to a large extent an open question. We discuss future directions of Numerical Question Answering and summarize guidelines on future dataset collection and system design.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

35 of 52 references.

References · 35

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  7. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  8. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  9. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  10. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  11. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  12. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
23 more
  1. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  2. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  3. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  4. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  5. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  6. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  7. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  8. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  9. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  10. 2022Reasoning Like Program Executorscited by 45
  11. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  12. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  13. 2019A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoningcited by 41
  14. 2021MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solverscited by 39
  15. 2020Question Directed Graph Attention Network for Numerical Reasoning over Textcited by 35
  16. 2020Semantically-Aligned Universal Tree-Structured Solver for Math Word Problemscited by 35
  17. 2018The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solverscited by 31
  18. 2016Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problemscited by 28
  19. 2021Adversarial Examples for Evaluating Math Word Problem Solverscited by 20
  20. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13
  21. 2021Numerical reasoning in machine reading comprehension tasks: are we there yet?cited by 9
  22. 2022Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solverscited by 9
  23. 2021Discrete Reasoning Templates for Natural Language Understandingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 2.

6 papers.

  1. 2025TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluationcited by 12
  2. 2024An Empirical Study of Data Ability Boundary in LLMs' Math Reasoningcited by 7
  3. 2025NumPert: Numerical Perturbations to Probe Language Models for Veracity Predictioncited by 1
  4. 2023Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoningcited by 0
  5. 2026PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Modelscited by 0
  6. 2026Towards Robust Numerical Claim Verificationcited by 0