# LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning

arXiv: https://arxiv.org/abs/2004.12817 · HTML: https://mlsanity.com/paper/2004.12817/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 25 of 25)

- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Compression of Generative Pre-trained Language Models via Quantization](https://mlsanity.com/paper/2203.10705/) (2022)
- [Black-Box On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2511.10643/) (2025)
- [NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search](https://mlsanity.com/paper/2105.14444/) (2021)
- [Direct Preference Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2406.19774/) (2024)
- [Structured Agent Distillation for Large Language Model](https://mlsanity.com/paper/2505.13820/) (2025)
- [Hybrid Policy Distillation for LLMs](https://mlsanity.com/paper/2604.20244/) (2026)
- [Generative Prompt Internalization](https://mlsanity.com/paper/2411.15927/) (2024)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.12947/) (2025)
- [FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices](https://mlsanity.com/paper/2410.00362/) (2024)
- [CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizers](https://mlsanity.com/paper/2502.16806/) (2025)
- [Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling](https://mlsanity.com/paper/2503.19123/) (2025)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [MoL for LLMs: Dual-Loss Optimization to Enhance Domain Expertise While Preserving General Capabilities](https://mlsanity.com/paper/2505.12043/) (2025)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [Language Model Distillation: A Temporal Difference Imitation Learning Perspective](https://mlsanity.com/paper/2505.20335/) (2025)
- [Anchored Sliding Window: Toward Robust and Imperceptible Linguistic Steganography](https://mlsanity.com/paper/2604.09066/) (2026)
- [ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages](https://mlsanity.com/paper/2407.03387/) (2024)
- [On Membership Inference Attacks in Knowledge Distillation](https://mlsanity.com/paper/2505.11837/) (2025)
- [Scheduling Optimization Techniques for Neural Network Training](https://mlsanity.com/paper/2110.00929/) (2021)
- [Decoder-based Sense Knowledge Distillation](https://mlsanity.com/paper/2602.22351/) (2026)
- [Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization](https://mlsanity.com/paper/2604.03110/) (2026)
- [PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation](https://mlsanity.com/paper/2606.23124/) (2026)
- [WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA](https://mlsanity.com/paper/2608.01035/) (2026)

## Abstract (arXiv)

While pre-training and fine-tuning, e.g., BERT~citepdevlin2018bert, GPT-2~citepradford2019language, have achieved great success in language understanding and generation tasks, the pre-trained models are usually too big for online deployment in terms of both memory cost and inference speed, which hinders them from practical online usage. In this paper, we propose LightPAFF, a Lightweight Pre-training And Fine-tuning Framework that leverages two-stage knowledge distillation to transfer knowledge from a big teacher model to a lightweight student model in both pre-training and fine-tuning stages. In this way the lightweight model can achieve similar accuracy as the big teacher model, but with much fewer parameters and thus faster online inference speed. LightPAFF can support different pre-training methods (such as BERT, GPT-2 and MASS~citepsong2019mass) and be applied to many downstream tasks. Experiments on three language understanding tasks, three language modeling tasks and three sequence to sequence generation tasks demonstrate that while achieving similar accuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.