mlsanityData to 14 Oct 2026

Basic Reading Distillation

arXiv 2507.19741 ↗First posted 26 Jul 2025cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable abilities in various natural language processing areas, but they demand high computation resources which limits their deployment in real-world. Distillation is one technique to solve this problem through either knowledge distillation or task distillation. Both distillation approaches train small models to imitate specific features of LLMs, but they all neglect basic reading education for small models on generic texts that are unrelated to downstream tasks. In this paper, we propose basic reading distillation (BRD) which educates a small model to imitate LLMs basic reading behaviors, such as named entity recognition, question raising and answering, on each sentence. After such basic education, we apply the small model on various tasks including language inference benchmarks and BIG-bench tasks. It shows that the small model can outperform or perform comparable to over 20x bigger LLMs. Analysis reveals that BRD effectively influences the probability distribution of the small model, and has orthogonality to either knowledge distillation or task distillation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 25 of its 36 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

25 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes23 Jun

Other references · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  10. 2022Large Language Models Are Reasoning Teacherscited by 329
  11. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 292
  12. 2022Teaching Small Language Models to Reasoncited by 284
12 more
  1. 2023The False Promise of Imitating Proprietary LLMscited by 263
  2. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  3. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  4. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  5. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
  6. 2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 62
  7. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  8. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  9. 2023Pre-Training to Learn in Contextcited by 42
  10. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  11. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  12. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  2. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  3. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  4. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
  5. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  6. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation
  7. 2023MiniLLM: On-Policy Distillation of Large Language Models
  8. 2023A Cohesive Distillation Architecture for Neural Language Models