# Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net

arXiv: https://arxiv.org/abs/1708.04106 · HTML: https://mlsanity.com/paper/1708.04106/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2017.

## Built on by (most cited first, 27 of 27)

- [Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation](https://mlsanity.com/paper/2107.01378/) (2021)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [FM2: Field-matrixed Factorization Machines for Recommender Systems](https://mlsanity.com/paper/2102.12994/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Network Augmentation for Tiny Deep Learning](https://mlsanity.com/paper/2110.08890/) (2021)
- [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [Supervision Complexity and its Role in Knowledge Distillation](https://mlsanity.com/paper/2301.12245/) (2023)
- [Lifelong Teacher-Student Network Learning](https://mlsanity.com/paper/2107.04689/) (2021)
- [Concept-based Explanations for Out-Of-Distribution Detectors](https://mlsanity.com/paper/2203.02586/) (2022)
- [User Response Prediction in Online Advertising](https://mlsanity.com/paper/2101.02342/) (2021)
- [DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation](https://mlsanity.com/paper/2505.19504/) (2025)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)
- [Knowledge Distillation Meets Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2205.06701/) (2022)
- [Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation](https://mlsanity.com/paper/2209.15560/) (2022)
- [MC2-SF: Slow-Fast Learning for Mobile-Cloud Collaborative Recommendation](https://mlsanity.com/paper/2109.12314/) (2021)
- [Can pruning make Large Language Models more efficient?](https://mlsanity.com/paper/2310.04573/) (2023)
- [Computation Resource Allocation Solution in Recommender Systems](https://mlsanity.com/paper/2103.02259/) (2021)
- [Audio Representation Learning by Distilling Video as Privileged Information](https://mlsanity.com/paper/2302.02845/) (2023)
- [Knowledge Distillation from Single to Multi Labels: an Empirical Study](https://mlsanity.com/paper/2303.08360/) (2023)
- [Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation](https://mlsanity.com/paper/2211.11159/) (2022)
- [NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants](https://mlsanity.com/paper/2306.13586/) (2023)
- [Collaborative Knowledge Distillation via a Learning-by-Education Node Community](https://mlsanity.com/paper/2410.00074/) (2024)
- [Beyond Student: An Asymmetric Network for Neural Network Inheritance](https://mlsanity.com/paper/2602.09509/) (2026)
- [Multi scale Feature Extraction and Fusion for Online Knowledge Distillation](https://mlsanity.com/paper/2206.08224/) (2022)
- [AccEPT: An Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training](https://mlsanity.com/paper/2311.05827/) (2023)

## Abstract (arXiv)

Models applied on real time response task, like click-through rate (CTR) prediction model, require high accuracy and rigorous response time. Therefore, top-performing deep models of high depth and complexity are not well suited for these applications with the limitations on the inference time. In order to further improve the neural networks' performance given the time and computational limitations, we propose an approach that exploits a cumbersome net to help train the lightweight net for prediction. We dub the whole process rocket launching, where the cumbersome booster net is used to guide the learning of the target light net throughout the whole training process. We analyze different loss functions aiming at pushing the light net to behave similarly to the booster net, and adopt the loss with best performance in our experiments. We use one technique called gradient block to improve the performance of the light net and booster net further. Experiments on benchmark datasets and real-life industrial advertisement data present that our light model can get performance only previously achievable with more complex models.