# Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks arXiv: https://arxiv.org/abs/2404.13099 ยท HTML: https://mlsanity.com/paper/2404.13099/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 33 references that are in the map. - First posted on arXiv 19 Apr 2024. ## Builds on (most cited first, 17 of 17) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021) - [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023) - [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022) - [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022) - [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019) - [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023) - [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023) - [NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://mlsanity.com/paper/2204.05660/) (2022) - [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021) - [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023) - [Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning](https://mlsanity.com/paper/2108.06743/) (2021) - [Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition](https://mlsanity.com/paper/2304.10977/) (2023) - [Enhancing LLM with Evolutionary Fine Tuning for News Summary Generation](https://mlsanity.com/paper/2307.02839/) (2023) - [Generative AI: Implications and Applications for Education](https://mlsanity.com/paper/2305.07605/) (2023) - [KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models](https://mlsanity.com/paper/2404.09763/) (2024) ## Built on by (most cited first, 12 of 12) - [Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents](https://mlsanity.com/paper/2412.00821/) (2024) - [Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges](https://mlsanity.com/paper/2502.08680/) (2025) - [Transforming Wearable Data into Personal Health Insights using Large Language Model Agents](https://mlsanity.com/paper/2406.06464/) (2024) - [When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning](https://mlsanity.com/paper/2505.15276/) (2025) - [Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback](https://mlsanity.com/paper/2412.06827/) (2024) - [Context-Enhanced Language Models for Generating Multi-Paper Citations](https://mlsanity.com/paper/2404.13865/) (2024) - [Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities](https://mlsanity.com/paper/2407.06125/) (2024) - [GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2607.13248/) (2026) - [Steps are all you need: Rethinking STEM Education with Prompt Engineering](https://mlsanity.com/paper/2412.05023/) (2024) - [GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models](https://mlsanity.com/paper/2508.03737/) (2025) - [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025) - [Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models](https://mlsanity.com/paper/2412.01353/) (2024) ## Abstract (arXiv) The rapid progress in the field of natural language processing (NLP) systems and the expansion of large language models (LLMs) have opened up numerous opportunities in the field of education and instructional methods. These advancements offer the potential for tailored learning experiences and immediate feedback, all delivered through accessible and cost-effective services. One notable application area for this technological advancement is in the realm of solving mathematical problems. Mathematical problem-solving not only requires the ability to decipher complex problem statements but also the skill to perform precise arithmetic calculations at each step of the problem-solving process. However, the evaluation of the arithmetic capabilities of large language models remains an area that has received relatively little attention. In response, we introduce an extensive mathematics dataset called "MathQuest" sourced from the 11th and 12th standard Mathematics NCERT textbooks. This dataset encompasses mathematical challenges of varying complexity and covers a wide range of mathematical concepts. Utilizing this dataset, we conduct fine-tuning experiments with three prominent LLMs: LLaMA-2, WizardMath, and MAmmoTH. These fine-tuned models serve as benchmarks for evaluating their performance on our dataset. Our experiments reveal that among the three models, MAmmoTH-13B emerges as the most proficient, achieving the highest level of competence in solving the presented mathematical problems. Consequently, MAmmoTH-13B establishes itself as a robust and dependable benchmark for addressing NCERT mathematics problems.