mlsanityData to 7 Oct 2026

Explanations from Large Language Models Make Small Reasoners Better

arXiv 2210.06726 ↗First posted 13 Oct 2022cs.CLcited by 123

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations. In this paper, we consider the problem of leveraging the explanations generated by LLM to improve the training of small reasoners, which are more favorable in real-production deployment due to their low cost. We systematically explore three explanation generation approaches from LLM and utilize a multi-task learning framework to facilitate small models to acquire strong reasoning power together with explanation generation capabilities. Experiments on multiple reasoning tasks show that our method can consistently and significantly outperform finetuning baselines across different settings, and even perform better than finetuning/prompting a 60x larger GPT-3 (175B) model by up to 9.5% in accuracy. As a side benefit, human evaluation further shows that our method can generate high-quality explanations to justify its predictions, moving towards the goal of explainable AI.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 54 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

30 of 54 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
18 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  6. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  7. 2022Can language models learn from explanations in context?cited by 248
  8. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  9. 2016Generating Visual Explanationscited by 156
  10. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  11. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  12. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  13. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  14. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  15. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  16. 2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
  17. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  18. 2019Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approachcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 36; 2024: 50; 2025: 23; 2026: 7.

123 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  6. 2022Large Language Models Are Reasoning Teacherscited by 318
  7. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  8. 2022Teaching Small Language Models to Reasoncited by 279
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2023A Survey on Model Compression for Large Language Modelscited by 263
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  12. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  3. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  4. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  5. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  6. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  7. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 103
  8. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  9. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  10. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  11. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  12. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  13. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  14. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  15. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  16. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  17. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  18. 2024A Survey on Transformer Compressioncited by 44
  19. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  20. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  21. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  22. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  23. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  24. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  25. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  26. 2023Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generationcited by 27
  27. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  28. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  29. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  30. 2024MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoningcited by 22
  31. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  32. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  33. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  34. 2024Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Trainingcited by 17
  35. 2023CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibrationcited by 16
  36. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15
  37. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  38. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
  2. 2023OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models
  3. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  4. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs
  5. 2023TART: A plug-and-play Transformer module for task-agnostic reasoning
  6. 2025Reasoning-Grounded Natural Language Explanations for Language Models
  7. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  8. 2022Complementary Explanations for Effective In-Context Learning