mlsanityData to 7 Oct 2026

Orca 2: Teaching Small Language Models How to Reason

arXiv 2311.11045 ↗First posted 18 Nov 2023cs.AIcited by 157

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Orca 1 learns from rich signals, such as explanation traces, allowing it to outperform conventional instruction-tuned models on benchmarks like BigBench Hard and AGIEval. In Orca 2, we continue exploring how improved training signals can enhance smaller LMs' reasoning abilities. Research on training small LMs has often relied on imitation learning to replicate the output of more capable models. We contend that excessive emphasis on imitation may restrict the potential of smaller models. We seek to teach small LMs to employ different solution strategies for different tasks, potentially different from the one used by the larger model. For example, while larger models might provide a direct answer to a complex task, smaller models may not have the same capacity. In Orca 2, we teach the model various reasoning techniques (step-by-step, recall then generate, recall-reason-generate, direct answer, etc.). More crucially, we aim to help the model learn to determine the most effective solution strategy for each task. We evaluate Orca 2 using a comprehensive set of 15 diverse benchmarks (corresponding to approximately 100 tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of similar size and attains performance levels similar or better to those of models 5-10x larger, as assessed on complex tasks that test advanced reasoning abilities in zero-shot settings. make Orca 2 weights publicly available at aka.ms/orca-lm to support research on the development, evaluation, and alignment of smaller LMs

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

34 of 69 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
22 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  11. 2023The False Promise of Imitating Proprietary LLMscited by 254
  12. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  13. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  14. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  15. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  16. 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
  17. 2023Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratingscited by 33
  18. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  19. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  20. 2022Lila: A Unified Benchmark for Mathematical Reasoningcited by 13
  21. 2023A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applicationscited by 12
  22. 2021Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performancecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 87; 2025: 46; 2026: 21.

157 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  3. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  5. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  6. 2025Phi-4-reasoning Technical Reportcited by 146
  7. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  8. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  9. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  10. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  11. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  12. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
38 more
  1. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  2. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  3. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  4. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  5. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  6. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  7. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Modelscited by 38
  8. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  9. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  10. 2024TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identificationcited by 33
  11. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  12. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  13. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  14. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  15. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  16. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  17. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  18. 2024HDLdebugger: Streamlining HDL debugging with Large Language Modelscited by 25
  19. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 25
  20. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  21. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  22. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  23. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  24. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  25. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  26. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  27. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 22
  28. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  29. 2024Towards Neural Synthesis for SMT-Assisted Proof-Oriented Programmingcited by 19
  30. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  31. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 17
  32. 2024Reformatted Alignmentcited by 15
  33. 2025TigerLLM - A Family of Bangla Large Language Modelscited by 15
  34. 2024AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimationcited by 14
  35. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  36. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  37. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoningcited by 13
  38. 2024textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuningcited by 13

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
  2. 2023Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization
  3. 2023Teaching Language Models to Self-Improve through Interactive Demonstrations
  4. 2025Big Reasoning with Small Models: Instruction Retrieval at Inference Time
  5. 2022Teaching Small Language Models to Reason
  6. 2025Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
  7. 2025Small Models Struggle to Learn from Strong Reasoners
  8. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation