mlsanityData to 7 Oct 2026

Adversarial Examples for Evaluating Math Word Problem Solvers

arXiv 2109.05925 ↗First posted 13 Sep 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40 percentage points on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

16 of 33 references.

References · 16

  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  8. 2018Generating Natural Language Adversarial Examplescited by 422
  9. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  10. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  11. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  12. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
4 more
  1. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  2. 2020Generating Natural Language Attacks in a Hard Label Black Box Settingcited by 50
  3. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  4. 2020A Context Aware Approach for Generating Natural Language Attackscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 9; 2024: 3; 2025: 1.

20 papers.

Sort:

By area:

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  3. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  4. 2024SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphscited by 31
  5. 2023Learning by Analogy: Diverse Questions Generation in Math Word Problemcited by 18
  6. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  7. 2025TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluationcited by 11
  8. 2022Self-consistent Reasoning For Solving Math Word Problemscited by 10
  9. 2022Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solverscited by 9
  10. 2022Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solverscited by 9
  11. 2022Unbiased Math Word Problems Benchmark for Mitigating Solving Biascited by 8
  12. 2023Math Word Problem Solving by Generating Linguistic Variants of Problem Statementscited by 7
8 more
  1. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6
  2. 2023Expression Syntax Information Bottleneck for Math Word Problemscited by 4
  3. 2022Improving Compositional Generalization in Math Word Problem Solvingcited by 3
  4. 2023FERMAT: An Alternative to Accuracy for Numerical Reasoningcited by 3
  5. 2022Textual Enhanced Contrastive Learning for Solving Math Word Problemscited by 1
  6. 2023Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoningcited by 0
  7. 2023Explaining Math Word Problem Solverscited by 0
  8. 2023From text to multimodal: a survey of adversarial example generation in question answering systemscited by 0