mlsanityData to 7 Oct 2026

Orca-Math: Unlocking the potential of SLMs in Grade School Math

arXiv 2402.14830 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 151

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Mathematical word problem-solving has long been recognized as a complex task for small language models (SLMs). A recent study hypothesized that the smallest model size, needed to achieve over 80% accuracy on the GSM8K benchmark, is 34 billion parameters. To reach this level of performance with smaller models, researcher often train SLMs to generate Python code or use tools to help avoid calculation errors. Additionally, they employ ensembling, where outputs of up to 100 model runs are combined to arrive at a more accurate result. Result selection is done using consensus, majority vote or a separate a verifier model used in conjunction with the SLM. Ensembling provides a substantial boost in accuracy but at a significant cost increase with multiple calls to the model (e.g., Phi-GSM uses top-48 to boost the performance from 68.2 to 81.5). In this work, we present Orca-Math, a 7-billion-parameter SLM based on the Mistral-7B, which achieves 86.81% on GSM8k without the need for multiple model calls or the use of verifiers, code execution or any other external tools. Our approach has the following key elements: (1) A high quality synthetic dataset of 200K math problems created using a multi-agent setup where agents collaborate to create the data, (2) An iterative learning techniques that enables the SLM to practice solving problems, receive feedback on its solutions and learn from preference pairs incorporating the SLM solutions and the feedback. When trained with Supervised Fine-Tuning alone, Orca-Math achieves 81.50% on GSM8k pass@1 metric. With iterative preference learning, Orca-Math achieves 86.81% pass@1. Orca-Math surpasses the performance of significantly larger models such as LLAMA-2-70B, WizardMath-70B, Gemini-Pro, ChatGPT-3.5. It also significantly outperforms other smaller models while using much smaller data (hundreds of thousands vs. millions of problems).

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 38 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

25 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023TinyGSM: achieving >80% on GSM8k with small language models14 Dec
  2. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning16 Nov
  3. 2023Llemma: An Open Language Model For Mathematics16 Oct
  4. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  6. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks12 Apr

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  7. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2024Self-Rewarding Language Modelscited by 654
  11. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
7 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  3. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  4. 2016Solving General Arithmetic Word Problemscited by 353
  5. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  6. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  7. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 67; 2026: 31.

151 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2024What matters when building vision-language models?cited by 321
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  7. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  10. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  11. 2025Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Provingcited by 126
  12. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
38 more
  1. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  2. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  4. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  5. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  6. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  7. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  8. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  9. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 79
  10. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  11. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  12. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  13. 2025AnyEdit: Edit Any Knowledge Encoded in Language Modelscited by 74
  14. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  15. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  16. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  17. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  18. 2025LLM Agents for Education: Advances and Applicationscited by 59
  19. 2024Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoningcited by 52
  20. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  21. 2025FineVision: Open Data Is All You Needcited by 51
  22. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  23. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  24. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  25. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  26. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  27. 2025Vision as LoRAcited by 37
  28. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  29. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  30. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  31. 2024Weak-to-Strong Reasoningcited by 32
  32. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  33. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  34. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  35. 2025Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluationcited by 29
  36. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  37. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  38. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  2. 2024Common 7B Language Models Already Possess Strong Math Capabilities
  3. 2025Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
  4. 2025EasyMath: A 0-shot Math Benchmark for SLMs
  5. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
  6. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
  7. 2024Small Language Models: Survey, Measurements, and Insights
  8. 2025Evaluation of LLMs for mathematical problem solving