mlsanityData to 14 Oct 2026

Mathematical Capabilities of ChatGPT

arXiv 2301.13867 ↗First posted 31 Jan 2023cs.LG cs.AI cs.CLcited by 268

Not on the map. Why?

Cited by 65 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!

Key facts. Cited by 268 papers in the map (data through 14 Oct 2026). Builds on 25 of its 101 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

25 of 101 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2018Datasheets for Datasetscited by 1,386
13 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2018Model Cards for Model Reportingcited by 1,080
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  5. 2019Deep Learning for Symbolic Mathematicscited by 270
  6. 2019The Lean mathematical librarycited by 134
  7. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  8. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
  9. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  10. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  11. 2023Mathematics, word problems, common sense, and artificial intelligencecited by 12
  12. 2022Machine Learning Class Numbers of Real Quadratic Fieldscited by 6
  13. 2020Mathematical Reasoning via Self-supervised Skip-tree Trainingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 105; 2024: 76; 2025: 62; 2026: 25.

268 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  7. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  8. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  9. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  10. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  11. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  12. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
38 more
  1. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  2. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  3. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  4. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  5. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  6. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  8. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
  9. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  10. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  11. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  13. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  14. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  15. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  16. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
  17. 2024Large Language Models and Games: A Survey and Roadmapcited by 93
  18. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  19. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  20. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  21. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  22. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 79
  23. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  24. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 77
  25. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  26. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
  27. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  28. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  29. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  30. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  31. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 68
  32. 2023ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Designcited by 66
  33. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  34. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  35. 2023Can we trust the evaluation on ChatGPT?cited by 63
  36. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  37. 2024A Survey on Deep Learning for Theorem Provingcited by 57
  38. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55

50 most cited of 268 All 268 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Cross-Language Assessment of Mathematical Capability of ChatGPT
  2. 2023Measuring reasoning capabilities of ChatGPT
  3. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  4. 2025Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
  5. 2026The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?
  6. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
  7. 2023ChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questions
  8. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning