mlsanityData to 7 Oct 2026

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

arXiv 2505.12781 ↗First posted 19 May 2025cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 64 of its 74 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

64 of 74 references.

References · 64

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  13. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  14. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  15. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  16. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  17. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  18. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  20. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  21. 2016Sequence-Level Knowledge Distillationcited by 761
  22. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  23. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  24. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  25. 2023Instruction Tuning with GPT-4cited by 676
  26. 2024Phi-4 Technical Reportcited by 675
  27. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  28. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  29. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  30. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  31. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  32. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  33. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  34. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  35. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  36. 2024InternLM2 Technical Reportcited by 391
  37. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  38. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2026EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discoverycited by 41
  2. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  3. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  4. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  5. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  6. 2026Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapsecited by 4
  7. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3
  8. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3
  9. 2026Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initializationcited by 2
  10. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  11. 2025Estimating the Effective Rank of Vision Transformers via Low-Rank Factorizationcited by 1
  12. 2025Projected Compression: Trainable Projection for Efficient Transformer Compressioncited by 0
3 more
  1. 2026UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillationcited by 0
  2. 2026Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillationcited by 0
  3. 2026FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networkscited by 0