mlsanityData to 7 Oct 2026

An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)

arXiv 2302.13814 ↗First posted 23 Feb 2023cs.CL cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 8 of its 18 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

8 of 18 references.

References · 8

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2021MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solverscited by 39
  6. 2016Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problemscited by 28
  7. 2017Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Carscited by 28
  8. 2016Introspective Perception: Learning to Predict Failures in Vision Systemscited by 17

Built on by

Cited by year

Citing papers per year: 2023: 15; 2024: 14; 2025: 2.

31 papers.

Sort:

By area:

  1. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  2. 2023How is ChatGPT's behavior changing over time?cited by 265
  3. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  4. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  5. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  6. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  7. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  8. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  9. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  10. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  11. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
  12. 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 31
19 more
  1. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
  2. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  3. 2024Neuro-Symbolic Data Generation for Math Reasoningcited by 21
  4. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  5. 2023ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Timecited by 12
  6. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  7. 2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughtscited by 8
  8. 2024Exploring Failure Cases in Multimodal Reasoning About Physical Dynamicscited by 5
  9. 2023Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queriescited by 4
  10. 2023Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bardcited by 3
  11. 2023Evaluating and Enhancing Large Language Models for Conversational Reasoning on Knowledge Graphscited by 3
  12. 2024Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventionscited by 3
  13. 2024Physics-Informed Autonomous LLM Agents for Explainable Power Electronics Modulation Designcited by 3
  14. 2024When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problemscited by 2
  15. 2024Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generationcited by 2
  16. 2023ChatGPT and post-test probabilitycited by 1
  17. 2025Error Detection and Correction for Interpretable Mathematics in Large Language Modelscited by 1
  18. 2023Matrix diagonalization and singular value decomposition: Static SageMath and dynamic ChatGPT juxtaposedcited by 0
  19. 2023ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Examscited by 0