mlsanityData to 7 Oct 2026

Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA

arXiv 2308.04679 ↗First posted 9 Aug 2023cs.CL cs.AIcited by 13

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large Language Models (LLMs) have shown outstanding performance across wide range of downstream tasks. This competency is attributed to their substantial parameter size and pre-training on extensive corpus. Moreover, LLMs have exhibited enhanced reasoning capabilities in tackling complex reasoning tasks, owing to the utilization of a method named ``Chain-of-Thought (CoT) prompting''. This method is designed to generate intermediate reasoning steps that guide the inference of the final answer. However, it is essential to highlight that these advanced reasoning abilities appear to emerge in models with a minimum of 10 billion parameters, thereby limiting its efficacy in situations where computational resources are constrained. In this paper, we investigate the possibility of transferring the reasoning capabilities of LLMs to smaller models via knowledge distillation. Specifically, we propose Sci-CoT, a two-stage framework that separates the processes of generating rationales and inferring answers. This method enables a more efficient use of rationales during the answer inference stage, leading to improved performance on scientific question-answering tasks. Utilizing Sci-CoT, our 80-million parameter model is able to exceed the performance of BLOOM-176B in the ARC-Easy dataset under the few shot setting.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 19 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

19 of 19 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Large Language Models Are Reasoning Teachers20 Dec
  2. 2022Teaching Small Language Models to Reason16 Dec
  3. 2022Large Language Models are Zero-Shot Reasoners24 May
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
3 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  2. 2023Low-Parameter Federated Learning with Large Language Modelscited by 22
  3. 2023Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answeringcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 8; 2025: 2.

13 papers.

Sort:

By area:

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Toolcited by 35
  3. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  4. 2024Learning to Maximize Mutual Information for Chain-of-Thought Distillationcited by 23
  5. 2024Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfercited by 19
  6. 2024Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thoughtcited by 12
  7. 2023On the Opportunities of Green Computing: A Surveycited by 7
  8. 2024From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervisioncited by 2
  9. 2025StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallelcited by 2
  10. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainabilitycited by 2
  11. 2024AS-ES Learning: Towards Efficient CoT Learning in Small Modelscited by 1
  12. 2023Two Timin': Repairing Smart Contracts With A Two-Layered Approachcited by 0
1 more
  1. 2024Can only LLMs do Reasoning?: Potential of Small Language Models in Task Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  2. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering
  3. 2025CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning
  4. 2022Distilling Reasoning Capabilities into Smaller Language Models
  5. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
  6. 2023MCC-KD: Multi-CoT Consistent Knowledge Distillation
  7. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering
  8. 2025Early Stopping Chain-of-thoughts in Large Language Models