# ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning

arXiv: https://arxiv.org/abs/2410.19056 · HTML: https://mlsanity.com/paper/2410.19056/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [Evaluating Mathematical Reasoning Beyond Accuracy](https://mlsanity.com/paper/2404.05692/) (2024)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [Conceptual and Unbiased Reasoning in Language Models](https://mlsanity.com/paper/2404.00205/) (2024)
- [FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation](https://mlsanity.com/paper/2406.11243/) (2024)

## Built on by (most cited first, 9 of 9)

- [RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning](https://mlsanity.com/paper/2505.14140/) (2025)
- [One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs](https://mlsanity.com/paper/2502.10454/) (2025)
- [Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities](https://mlsanity.com/paper/2509.25725/) (2025)
- [Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation](https://mlsanity.com/paper/2507.02253/) (2025)
- [BOW: Training Language Models to Reason Over Plausible Next Words](https://mlsanity.com/paper/2506.13502/) (2025)
- [Evaluating Program Semantics Reasoning with Type Inference in System F](https://mlsanity.com/paper/2509.23686/) (2025)
- [GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentations](https://mlsanity.com/paper/2605.07053/) (2026)
- [Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning](https://mlsanity.com/paper/2604.00018/) (2026)
- [CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning](https://mlsanity.com/paper/2512.18857/) (2025)

## Abstract (arXiv)

Existing math datasets evaluate the reasoning abilities of large language models (LLMs) by either using the final answer or the intermediate reasoning steps derived from static examples. However, the former approach fails to surface model's uses of shortcuts and wrong reasoning while the later poses challenges in accommodating alternative solutions. In this work, we seek to use symbolic programs as a means for automated evaluation if a model can consistently produce correct final answers across various inputs to the program. We begin by extracting programs for popular math datasets (GSM8K and MATH) using GPT4-o. For those executable programs verified using the original input-output pairs, they are found to encapsulate the proper reasoning required to solve the original text questions. We then prompt GPT4-o to generate new questions using alternative input-output pairs based the extracted program. We apply the resulting datasets to evaluate a collection of LLMs. In our experiments, we observe significant accuracy drops using our proposed evaluation compared with original static examples, suggesting the fragility of math reasoning in state-of-the-art LLMs.