mlsanityData to 14 Oct 2026

Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

arXiv 2404.13099 ↗First posted 19 Apr 2024cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The rapid progress in the field of natural language processing (NLP) systems and the expansion of large language models (LLMs) have opened up numerous opportunities in the field of education and instructional methods. These advancements offer the potential for tailored learning experiences and immediate feedback, all delivered through accessible and cost-effective services. One notable application area for this technological advancement is in the realm of solving mathematical problems. Mathematical problem-solving not only requires the ability to decipher complex problem statements but also the skill to perform precise arithmetic calculations at each step of the problem-solving process. However, the evaluation of the arithmetic capabilities of large language models remains an area that has received relatively little attention. In response, we introduce an extensive mathematics dataset called "MathQuest" sourced from the 11th and 12th standard Mathematics NCERT textbooks. This dataset encompasses mathematical challenges of varying complexity and covers a wide range of mathematical concepts. Utilizing this dataset, we conduct fine-tuning experiments with three prominent LLMs: LLaMA-2, WizardMath, and MAmmoTH. These fine-tuned models serve as benchmarks for evaluating their performance on our dataset. Our experiments reveal that among the three models, MAmmoTH-13B emerges as the most proficient, achieving the highest level of competence in solving the presented mathematical problems. Consequently, MAmmoTH-13B establishes itself as a robust and dependable benchmark for addressing NCERT mathematics problems.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

17 of 33 references.

References · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  8. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  9. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  10. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  11. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  12. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 68
5 more
  1. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17
  2. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  3. 2023Enhancing LLM with Evolutionary Fine Tuning for News Summary Generationcited by 10
  4. 2023Generative AI: Implications and Applications for Educationcited by 8
  5. 2024KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 1.

12 papers.

  1. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agentscited by 24
  2. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  3. 2024Transforming Wearable Data into Personal Health Insights using Large Language Model Agentscited by 19
  4. 2025When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoningcited by 7
  5. 2024Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedbackcited by 6
  6. 2024Context-Enhanced Language Models for Generating Multi-Paper Citationscited by 5
  7. 2024Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalitiescited by 5
  8. 2026GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Modelscited by 2
  9. 2024Steps are all you need: Rethinking STEM Education with Prompt Engineeringcited by 1
  10. 2025GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Modelscited by 1
  11. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  12. 2024Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Modelscited by 0