# Explaining Math Word Problem Solvers

arXiv: https://arxiv.org/abs/2307.13128 · HTML: https://mlsanity.com/paper/2307.13128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Adversarial Examples for Evaluating Math Word Problem Solvers](https://mlsanity.com/paper/2109.05925/) (2021)
- [Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers](https://mlsanity.com/paper/2205.00177/) (2022)
- [Solving Arithmetic Word Problems Automatically Using Transformer and Unambiguous Representations](https://mlsanity.com/paper/1912.00871/) (2019)
- [QADiver: Interactive Framework for Diagnosing QA Models](https://mlsanity.com/paper/1812.00161/) (2018)

## Abstract (arXiv)

Automated math word problem solvers based on neural networks have successfully managed to obtain 70-80% accuracy in solving arithmetic word problems. However, it has been shown that these solvers may rely on superficial patterns to obtain their equations. In order to determine what information math word problem solvers use to generate solutions, we remove parts of the input and measure the model's performance on the perturbed dataset. Our results show that the model is not sensitive to the removal of many words from the input and can still manage to find a correct answer when given a nonsense question. This indicates that automatic solvers do not follow the semantic logic of math word problems, and may be overfitting to the presence of specific words.