mlsanityData to 7 Oct 2026

Learning Multi-Step Reasoning by Solving Arithmetic Tasks

arXiv 2306.01707 ↗First posted 2 Jun 2023cs.CLcited by 12

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Mathematical reasoning is regarded as a necessary ability for Language Models (LMs). Recent works demonstrate large LMs' impressive performance in solving math problems. The success is attributed to their Chain-of-Thought (CoT) reasoning abilities, i.e., the ability to decompose complex questions into step-by-step reasoning chains, but such ability seems only to emerge from models with abundant parameters. This work investigates how to incorporate relatively small LMs with the capabilities of multi-step reasoning. We propose to inject such abilities by continually pre-training LMs on a synthetic dataset MsAT which is composed of Multi-step Arithmetic Tasks. Our experiments on four math word problem datasets show the effectiveness of the proposed method in enhancing LMs' math reasoning abilities.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 39 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

23 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction19 Mar

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
10 more
  1. 2022PAL: Program-aided Language Modelscited by 635
  2. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  3. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  4. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  5. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  6. 2022Reasoning Like Program Executorscited by 45
  7. 2021MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solverscited by 39
  8. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  9. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoningcited by 11
  10. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 1.

12 papers.

  1. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  2. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26
  3. 2024What Makes Math Word Problems Challenging for LLMs?cited by 19
  4. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  5. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  6. 2025LookAlike: Consistent Distractor Generation in Math MCQscited by 7
  7. 2023An Expression Tree Decoding Strategy for Mathematical Equation Generationcited by 4
  8. 2024Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventionscited by 3
  9. 2023FootGPT : A Large Language Model Development Experiment on a Minimal Settingcited by 1
  10. 2024Structured List-Grounded Question Answeringcited by 0
  11. 2025From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognitioncited by 0
  12. 2026The Power of Power Law: Asymmetry Enables Compositional Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Step Reasoning with Large Language Models, a Survey
  2. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics
  3. 2024Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning
  4. 2024Small Language Models are Equation Reasoners
  5. 2023Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
  6. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  7. 2024LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning
  8. 2025Large Language Models and Mathematical Reasoning Failures