# An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation

arXiv: https://arxiv.org/abs/2002.12597 · HTML: https://mlsanity.com/paper/2002.12597/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 14 of 14)

- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [TimeDistill: Efficient Long-Term Time Series Forecasting with MLP via Cross-Architecture Distillation](https://mlsanity.com/paper/2502.15016/) (2025)
- [Federated Distillation based Indoor Localization for IoT Networks](https://mlsanity.com/paper/2205.11440/) (2022)
- [Knowledge Distillation for Quality Estimation](https://mlsanity.com/paper/2107.00411/) (2021)
- [Synthetic data generation method for data-free knowledge distillation in regression neural networks](https://mlsanity.com/paper/2301.04338/) (2023)
- [Continual Learning in Predictive Autoscaling](https://mlsanity.com/paper/2307.15941/) (2023)
- [Knowledge boosting during low-latency inference](https://mlsanity.com/paper/2407.11055/) (2024)
- [Using Knowledge Distillation to improve interpretable models in a retail banking context](https://mlsanity.com/paper/2209.15496/) (2022)
- [Heterogeneous Transfer Learning for Building High-Dimensional Generalized Linear Models with Disparate Datasets](https://mlsanity.com/paper/2312.12786/) (2023)
- [Fast and accurate neural reflectance transformation imaging through knowledge distillation](https://mlsanity.com/paper/2510.24486/) (2025)
- [3D Reconstruction and Knowledge Distillation to Improve Multi-View Image Models to Explore Spike Volume Estimation in Wheat](https://mlsanity.com/paper/2605.20940/) (2026)

## Abstract (arXiv)

Compressing deep neural network (DNN) models becomes a very important and necessary technique for real-world applications, such as deploying those models on mobile devices. Knowledge distillation is one of the most popular methods for model compression, and many studies have been made on developing this technique. However, those studies mainly focused on classification problems, and very few attempts have been made on regression problems, although there are many application of DNNs on regression problems. In this paper, we propose a new formalism of knowledge distillation for regression problems. First, we propose a new loss function, teacher outlier rejection loss, which rejects outliers in training samples using teacher model predictions. Second, we consider a multi-task network with two outputs: one estimates training labels which is in general contaminated by noisy labels; And the other estimates teacher model's output which is expected to modify the noise labels following the memorization effects. By considering the multi-task network, training of the feature extraction of student models becomes more effective, and it allows us to obtain a better student model than one trained from scratch. We performed comprehensive evaluation with one simple toy model: sinusoidal function, and two open datasets: MPIIGaze, and Multi-PIE. Our results show consistent improvement in accuracy regardless of the annotation error level in the datasets.