mlsanityData to 14 Oct 2026

Scaling Laws for Task-Specific LLM Distillation

arXiv 2606.24747 ↗First posted 23 Jun 2026cs.AI cs.CEcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

31 of 46 references.

References · 31

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  12. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
19 more
  1. 2016Sequence-Level Knowledge Distillationcited by 801
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2019On the Efficacy of Knowledge Distillationcited by 381
  7. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  8. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2018Large scale distributed neural network training through online distillationcited by 146
  11. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  12. 2025Distillation Scaling Lawscited by 67
  13. 2019Is Deeper Better only when Shallow is Good?cited by 24
  14. 2020Why distillation helps: a statistical perspectivecited by 21
  15. 2024Large Language Model Adaptation for Financial Sentiment Analysiscited by 18
  16. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  17. 2023Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMscited by 13
  18. 2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
  19. 2024BERT vs GPT for financial engineeringcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  2. 2026UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation
  3. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  4. 2026Capability Scaling-Down Laws for LLM Compression
  5. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
  6. 2025On the Fundamental Limits of LLMs at Scale
  7. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
  8. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application