mlsanityData to 7 Oct 2026

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

arXiv 2401.09395 ↗First posted 17 Jan 2024cs.CLcited by 20

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMORE and HUMANEVAL-CORE, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks. Through comprehensive evaluations of both closed-source and open-source LLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology. We open-source the datasets and source codes at: https://github.com/declare-lab/LLM-ReasoningTest.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 51 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

26 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems3 Oct
  2. 2023ARB: Advanced Reasoning Benchmark for Large Language Models25 Jul
  3. 2023A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers21 May

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
11 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  3. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  4. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  5. 2022Language models show human-like content effects on reasoning taskscited by 171
  6. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 64
  7. 2021Self-Supervised Bug Detection and Repaircited by 59
  8. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  9. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  10. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoningcited by 22
  11. 2023CLOMO: Counterfactual Logical Modification with Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  3. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 81
  4. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  5. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  6. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
  7. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  8. 2026Large Language Model Reasoning Failurescited by 20
  9. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  10. 2024In Context Learning and Reasoning for Symbolic Regression with Large Language Modelscited by 7
  11. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 5
  12. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
8 more
  1. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoningcited by 1
  2. 2025ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluationcited by 1
  3. 2025Evaluating LLM Reasoning Beyond Correctness and CoTcited by 0
  4. 2025Decomposed Trust: Privacy, Adversarial Robustness, Ethics, and Fairness in Low-Rank LLMscited by 0
  5. 2026When Verification Fails: How Compositionally Infeasible Claims Escape Rejectioncited by 0
  6. 2026Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanismscited by 0
  7. 2026DiffPDE: Masked Diffusion Language Models as PDE Solvercited by 0
  8. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
  2. 2025Can LLMs textitunderstand Math? -- Exploring the Pitfalls in Mathematical Reasoning
  3. 2025Large Language Models and Mathematical Reasoning Failures
  4. 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges
  5. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  6. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  7. 2024MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs
  8. 2024An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning