mlsanityData to 7 Oct 2026

A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models

arXiv 2210.12023 ↗First posted 21 Oct 2022cs.CL cs.LGcited by 68

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We have recently witnessed a number of impressive results on hard mathematical reasoning problems with language models. At the same time, the robustness of these models has also been called into question; recent works have shown that models can rely on shallow patterns in the problem description when generating a solution. Building on the idea of behavioral testing, we propose a novel framework, which pins down the causal effect of various factors in the input, e.g., the surface form of the problem text, the operands, and math operators on the output solution. By grounding the behavioral analysis in a causal graph describing an intuitive reasoning process, we study the behavior of language models in terms of robustness and sensitivity to direct interventions in the input space. We apply our framework on a test bed of math word problems. Our analysis shows that robustness does not appear to continuously improve as a function of size, but the GPT-3 Davinci models (175B) achieve a dramatic improvement in both robustness and sensitivity compared to all other GPT variants.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 78 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

38 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks12 Apr

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
25 more
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  6. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  7. 2013Direct and Indirect Effectscited by 276
  8. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  9. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  10. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  11. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  12. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  13. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  14. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  15. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  16. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  17. 2020Do Language Embeddings Capture Scales?cited by 63
  18. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  19. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  20. 2020An Empirical Investigation of Contextualized Number Predictioncited by 21
  21. 2021Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLPcited by 21
  22. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  23. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13
  24. 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performancecited by 11
  25. 2021Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Contextcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 18; 2024: 27; 2025: 13; 2026: 7.

68 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  3. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  4. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  5. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 102
  6. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  7. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  8. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  9. 2023Autonomous GIS: the next-generation AI-powered GIScited by 52
  10. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  11. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  12. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 42
38 more
  1. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  2. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  3. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  4. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?cited by 36
  5. 2024Causal Inference with Large Language Model: A Surveycited by 31
  6. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  7. 2022Causally Testing Gender Bias in LLMs: A Case Study on Occupational Biascited by 28
  8. 2023Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibilitycited by 28
  9. 2024A Survey of Useful LLM Evaluationcited by 28
  10. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  11. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  12. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  13. 2024SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trialscited by 20
  14. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  15. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Modelscited by 19
  16. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  17. 2023NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reportscited by 18
  18. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  19. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  20. 2023A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformerscited by 15
  21. 2024FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answeringcited by 14
  22. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  23. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  24. 2023World Models for Math Story Problemscited by 11
  25. 2024How Well Do LLMs Identify Cultural Unity in Diversity?cited by 8
  26. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  27. 2025Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillationcited by 8
  28. 2025An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problemscited by 8
  29. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  30. 2024Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigationcited by 7
  31. 2025SteerX: Disentangled Steering for LLM Personalizationcited by 7
  32. 2023Multimodal Multi-Hop Question Answering Through a Conversation Between Tools and Efficiently Finetuned Large Language Modelscited by 6
  33. 2024How often are errors in natural language reasoning due to paraphrastic variability?cited by 5
  34. 2025JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulationcited by 5
  35. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  36. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  37. 2023Towards CausalGPT: A Multi-Agent Approach for Faithful Knowledge Reasoning via Promoting Causal Consistency in LLMscited by 4
  38. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Representation Robustness Under Executable Reasoning Constraints in Large Language Models for Mathematical Problem Solving
  2. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
  3. 2025Examining False Positives under Inference Scaling for Mathematical Reasoning
  4. 2025Toward Automated Robustness Evaluation of Mathematical Reasoning
  5. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
  6. 2025Large Language Models and Mathematical Reasoning Failures
  7. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
  8. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap