mlsanityData to 14 Oct 2026

Specializing Smaller Language Models towards Multi-Step Reasoning

arXiv 2301.12726 ↗First posted 30 Jan 2023cs.CL cs.AI cs.LGcited by 251

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The surprising ability of Large Language Models (LLMs) to perform well on complex reasoning with only few-shot chain-of-thought prompts is believed to emerge only in very large-scale models (100+ billion parameters). We show that such abilities can, in fact, be distilled down from GPT-3.5 (≥ 175B) to T5 variants (≤ 11B). We propose model specialization, to specialize the model's ability towards a target task. The hypothesis is that large models (commonly viewed as larger than 100B) have strong modeling power, but are spread on a large spectrum of tasks. Small models (commonly viewed as smaller than 10B) have limited model capacity, but if we concentrate their capacity on a specific target task, the model can achieve a decent improved performance. We use multi-step math reasoning as our testbed because it is a very typical emergent ability. We show two important aspects of model abilities: (1). there exists a very complex balance/ tradeoff between language models' multi-dimensional abilities; (2). by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability. We further give comprehensive discussions about important design choices for better generalization, including the tuning data format, the start model checkpoint, and a new model selection method. We hope our practice and discoveries can serve as an important attempt towards specialized smaller models in the new research paradigm set by LLMs.

Key facts. Cited by 251 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 31 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

22 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Explanations from Large Language Models Make Small Reasoners Better13 Oct

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
9 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2021MetaICL: Learning to Learn In Contextcited by 454
  4. 2022Large Language Models Are Reasoning Teacherscited by 329
  5. 2022Teaching Small Language Models to Reasoncited by 284
  6. 2022Prompting GPT-3 To Be Reliablecited by 260
  7. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  8. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  9. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 66; 2024: 97; 2025: 59; 2026: 29.

251 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  6. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  7. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  8. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  10. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  11. 2023A Survey on Model Compression for Large Language Modelscited by 269
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
38 more
  1. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 210
  2. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  3. 2023Efficient Large Language Models: A Surveycited by 186
  4. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  5. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  6. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  7. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  8. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  10. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  11. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 117
  12. 2024Knowledge Fusion of Large Language Modelscited by 115
  13. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  14. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  15. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
  16. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoningcited by 108
  17. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  18. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
  19. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  20. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  21. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  22. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  23. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  24. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  25. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  26. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  27. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  28. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  29. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  30. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 64
  31. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  32. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  33. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 63
  34. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 63
  35. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  36. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 60
  37. 2024TinyAgent: Function Calling at the Edgecited by 58
  38. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56

50 most cited of 251 All 251 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Capabilities and Invariability of Large Language Models
  2. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  4. 2025Bridging the Reasoning Gap: Small LLMs Can Plan with Generalised Strategies
  5. 2025Small Models Struggle to Learn from Strong Reasoners
  6. 2025Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them
  7. 2024Multi-Step Reasoning with Large Language Models, a Survey
  8. 2024Distilling LLMs' Decomposition Abilities into Compact Language Models