mlsanityData to 14 Oct 2026

Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks

arXiv 2305.18395 ↗First posted 28 May 2023cs.CL cs.AI cs.LGcited by 64

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge. However, deployment of the LLMs in real-world applications can be challenging due to their high computational requirements and concerns on data privacy. Previous studies have focused on building task-specific small Language Models (LMs) by fine-tuning them with labeled data or distilling LLMs. However, these approaches are ill-suited for knowledge-intensive reasoning tasks due to the limited capacity of small LMs in memorizing the knowledge required. Motivated by our theoretical analysis on memorization, we propose Knowledge-Augmented Reasoning Distillation (KARD), a novel method that fine-tunes small LMs to generate rationales obtained from LLMs with augmented knowledge retrieved from an external knowledge base. Moreover, we further propose a neural reranker to obtain documents relevant to rationale generation. We empirically show that KARD significantly improves the performance of small T5 and GPT models on the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE, StrategyQA, and OpenbookQA. Notably, our method makes the 250M T5 models achieve superior performance against the fine-tuned 3B models, having 12 times larger parameters, on both MedQA-USMLE and StrategyQA benchmarks.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 59 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

49 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Large Language Models Are Reasoning Teachers20 Dec
  2. 2022Teaching Small Language Models to Reason16 Dec
  3. 2022Large Language Models are Zero-Shot Reasoners24 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
32 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  13. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  14. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  15. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  16. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  17. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  18. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  19. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  20. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  21. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  22. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  23. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  24. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  25. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  26. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  27. 2022Can large language models reason about medical questions?cited by 194
  28. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  29. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 104
  30. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 78
  31. 2022Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Modelcited by 32
  32. 2022Variational Open-Domain Question Answeringcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 22; 2026: 15.

64 papers.

Sort:

By area:

  1. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  4. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  6. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  7. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
  8. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  9. 2023Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agentscited by 41
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  12. 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Modelscited by 34
38 more
  1. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  2. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  3. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  4. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  5. 2024Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosiscited by 23
  6. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
  7. 2025Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillationcited by 20
  8. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 19
  9. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  10. 2024I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responsescited by 15
  11. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  12. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  13. 2024Enhancing Q&A with Domain-Specific Fine-Tuning and Iterative Reasoning: A Comparative Studycited by 11
  14. 2026OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAGcited by 11
  15. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  16. 2024MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Modelscited by 7
  17. 2025FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusioncited by 6
  18. 2025Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysiscited by 6
  19. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  20. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  21. 2024Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Modelscited by 4
  22. 2024Achieving Sparse Activation in Small Language Modelscited by 4
  23. 2024Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Modelscited by 4
  24. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  25. 2026FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedbackcited by 4
  26. 2025Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuningcited by 3
  27. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  28. 2025The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Modelscited by 3
  29. 2026CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoningcited by 3
  30. 2024LAPIS: Language Model-Augmented Police Investigation Systemcited by 2
  31. 2024TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiencycited by 2
  32. 2024Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examplescited by 2
  33. 2025K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressorcited by 2
  34. 2025Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilitiescited by 2
  35. 2024Can only LLMs do Reasoning?: Potential of Small Language Models in Task Planningcited by 1
  36. 2024SleepCoT: A Lightweight Personalized Sleep Health Model via Chain-of-Thought Distillationcited by 1
  37. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexitycited by 1
  38. 2025It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisationcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
  2. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
  3. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  4. 2025Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMs
  5. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  6. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasoners
  7. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  8. 2025Small Models Struggle to Learn from Strong Reasoners