# Adversarial Examples for Evaluating Math Word Problem Solvers

arXiv: https://arxiv.org/abs/2109.05925 · HTML: https://mlsanity.com/paper/2109.05925/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [Generating Natural Language Attacks in a Hard Label Black Box Setting](https://mlsanity.com/paper/2012.14956/) (2020)
- [Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems](https://mlsanity.com/paper/1811.00720/) (2018)
- [A Context Aware Approach for Generating Natural Language Attacks](https://mlsanity.com/paper/2012.13339/) (2020)

## Built on by (most cited first, 20 of 20)

- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline](https://mlsanity.com/paper/2404.02893/) (2024)
- [MathAttack: Attacking Large Language Models Towards Math Solving Ability](https://mlsanity.com/paper/2309.01686/) (2023)
- [SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphs](https://mlsanity.com/paper/2410.02811/) (2024)
- [Learning by Analogy: Diverse Questions Generation in Math Word Problem](https://mlsanity.com/paper/2306.09064/) (2023)
- [Cutting Through the Noise: Boosting LLM Performance on Math Word Problems](https://mlsanity.com/paper/2406.15444/) (2024)
- [TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation](https://mlsanity.com/paper/2502.13442/) (2025)
- [Self-consistent Reasoning For Solving Math Word Problems](https://mlsanity.com/paper/2210.15373/) (2022)
- [Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers](https://mlsanity.com/paper/2205.00177/) (2022)
- [Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solvers](https://mlsanity.com/paper/2205.15683/) (2022)
- [Unbiased Math Word Problems Benchmark for Mitigating Solving Bias](https://mlsanity.com/paper/2205.08108/) (2022)
- [Math Word Problem Solving by Generating Linguistic Variants of Problem Statements](https://mlsanity.com/paper/2306.13899/) (2023)
- [Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems](https://mlsanity.com/paper/2211.07455/) (2022)
- [Expression Syntax Information Bottleneck for Math Word Problems](https://mlsanity.com/paper/2310.15664/) (2023)
- [Improving Compositional Generalization in Math Word Problem Solving](https://mlsanity.com/paper/2209.01352/) (2022)
- [FERMAT: An Alternative to Accuracy for Numerical Reasoning](https://mlsanity.com/paper/2305.17491/) (2023)
- [Textual Enhanced Contrastive Learning for Solving Math Word Problems](https://mlsanity.com/paper/2211.16022/) (2022)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [Explaining Math Word Problem Solvers](https://mlsanity.com/paper/2307.13128/) (2023)
- [From text to multimodal: a survey of adversarial example generation in question answering systems](https://mlsanity.com/paper/2312.16156/) (2023)

## Abstract (arXiv)

Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40 percentage points on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.