mlsanityData to 7 Oct 2026

Are NLP Models really able to Solve Simple Math Word Problems?

arXiv 2103.07191 ↗First posted 12 Mar 2021cs.CLcited by 861

Not on the map. Why?

Cited by 404 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

Key facts. Cited by 861 papers in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 12 Mar 2021.

Builds on

16 of 28 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2019Attention is not not Explanationcited by 629
  10. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  11. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  12. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
4 more
  1. 2017Mapping to Declarative Knowledge for Word Problem Solvingcited by 36
  2. 2020Semantically-Aligned Universal Tree-Structured Solver for Math Word Problemscited by 35
  3. 2018The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solverscited by 31
  4. 2020Exposing Shallow Heuristics of Relation Extraction Models with Challenge Datacited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 46; 2023: 171; 2024: 248; 2025: 226; 2026: 164.

861 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
38 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  2. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2022PAL: Program-aided Language Modelscited by 635
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  8. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  9. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  13. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  14. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  15. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  16. 2023Language Models can Solve Computer Taskscited by 371
  17. 2024Automated Design of Agentic Systemscited by 360
  18. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  19. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  20. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  21. 2022Large Language Models Are Reasoning Teacherscited by 318
  22. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  23. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  24. 2022UL2: Unifying Language Learning Paradigmscited by 292
  25. 2023Faithful Chain-of-Thought Reasoningcited by 285
  26. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  27. 2022Teaching Small Language Models to Reasoncited by 279
  28. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  29. 2022Generating Sequences by Learning to Self-Correctcited by 271
  30. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  31. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  32. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  33. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  34. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  35. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  36. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  37. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  38. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203

50 most cited of 861 All 861 (JSON)