Small Language Models are Equation Reasoners
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
Chain-of-Thought (CoT) reasoning has enabled Large Language Model (LLM) to achieve remarkable performance in various NLP tasks, including arithmetic problem-solving. However, this success does not generalize to small language model (sLM) like T5, due to their limited capacity and absence of emergent abilities associated with larger models. Recent works to enhance sLM through knowledge distillation have yielded some improvements but still face significant limitations, particularly high ambiguity from the variability in natural language expressions and substantial computational costs. In this paper, we investigate why sLM perform poorly on arithmetic reasoning tasks and hypothesize that natural language format variability introduces high ambiguity for these smaller models. Based on this hypothesis, we conduct experiments with equation-only format, which is a reasoning format that unifies arithmetic reasoning previously expressed in natural language formats into mathematical equations. Experiment results demonstrate that equation-only format effectively boosts the arithmetic reasoning abilities of sLM, especially in very small models like T5-Tiny.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 19 references that are in the map. First posted on arXiv 19 Sep 2024.
Builds on
19 of 19 references.
Closest ancestors · 1
Same direction, newest first
Other references · 18
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
- 2023PaLM 2 Technical Reportcited by 1,162
6 more
- 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2022Large Language Models Are Reasoning Teacherscited by 329
- 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
- 2024A Survey on Efficient Inference for Large Language Modelscited by 205
- 2024Distilling Mathematical Reasoning Capabilities into Small Language Modelscited by 14
Built on by
Citing papers per year: 2025: 2.
2 papers.
- 2025Towards Reasoning Ability of Small Language Modelscited by 29
- 2025EasyMath: A 0-shot Math Benchmark for SLMscited by 0
- 2025EasyMath: A 0-shot Math Benchmark for SLMs20 May 2025
- 2025Towards Reasoning Ability of Small Language Models17 Feb 2025
Similar, not cited
Close in wording, no citation either way
- 2023Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
- 2023Learning Multi-Step Reasoning by Solving Arithmetic Tasks
- 2025Large Language Models and Mathematical Reasoning Failures
- 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
- 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
- 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
- 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
- 2024Large Language Models for Mathematical Reasoning: Progresses and Challenges