# Cross-Task Knowledge Distillation in Multi-Task Recommendation

arXiv: https://arxiv.org/abs/2202.09852 · HTML: https://mlsanity.com/paper/2202.09852/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Self-Supervised Generalisation with Meta Auxiliary Learning](https://mlsanity.com/paper/1901.08933/) (2019)
- [Understanding and Improving Knowledge Distillation](https://mlsanity.com/paper/2002.03532/) (2020)
- [Self-Distillation as Instance-Specific Label Smoothing](https://mlsanity.com/paper/2006.05065/) (2020)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [Explainable Recommendation via Multi-Task Learning in Opinionated Text Data](https://mlsanity.com/paper/1806.03568/) (2018)
- [Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net](https://mlsanity.com/paper/1708.04106/) (2017)
- [Preparing Lessons: Improve Knowledge Distillation with Better Supervision](https://mlsanity.com/paper/1911.07471/) (2019)
- [Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising](https://mlsanity.com/paper/1907.10235/) (2019)
- [Privileged Features Distillation at Taobao Recommendations](https://mlsanity.com/paper/1907.05171/) (2019)
- [Ensemble Knowledge Distillation for CTR Prediction](https://mlsanity.com/paper/2011.04106/) (2020)

## Built on by (most cited first, 22 of 22)

- [AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning](https://mlsanity.com/paper/2211.15055/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks](https://mlsanity.com/paper/2210.13014/) (2022)
- [Instruction Tuned Models are Quick Learners](https://mlsanity.com/paper/2306.05539/) (2023)
- [STEM: Unleashing the Power of Embeddings for Multi-task Recommendation](https://mlsanity.com/paper/2308.13537/) (2023)
- [Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts](https://mlsanity.com/paper/2501.15065/) (2025)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)
- [Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation](https://mlsanity.com/paper/2402.02855/) (2024)
- [TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving](https://mlsanity.com/paper/2506.18084/) (2025)
- [Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolution](https://mlsanity.com/paper/2407.07302/) (2024)
- [Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning](https://mlsanity.com/paper/2507.21049/) (2025)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication](https://mlsanity.com/paper/2310.03188/) (2023)
- [Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System](https://mlsanity.com/paper/2404.15678/) (2024)
- [From Dense to Sparse: Event Response for Enhanced Residential Load Forecasting](https://mlsanity.com/paper/2501.02781/) (2025)
- [An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation](https://mlsanity.com/paper/2401.06356/) (2024)
- [Learning to Project for Cross-Task Knowledge Distillation](https://mlsanity.com/paper/2403.14494/) (2024)
- [Knowledge Migration Framework for Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2412.11175/) (2024)
- [VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding](https://mlsanity.com/paper/2510.07760/) (2025)
- [RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection](https://mlsanity.com/paper/2603.07493/) (2026)
- [Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression](https://mlsanity.com/paper/2608.00129/) (2026)

## Abstract (arXiv)

Multi-task learning (MTL) has been widely used in recommender systems, wherein predicting each type of user feedback on items (e.g, click, purchase) are treated as individual tasks and jointly trained with a unified model. Our key observation is that the prediction results of each task may contain task-specific knowledge about user's fine-grained preference towards items. While such knowledge could be transferred to benefit other tasks, it is being overlooked under the current MTL paradigm. This paper, instead, proposes a Cross-Task Knowledge Distillation framework that attempts to leverage prediction results of one task as supervised signals to teach another task. However, integrating MTL and KD in a proper manner is non-trivial due to several challenges including task conflicts, inconsistent magnitude and requirement of synchronous optimization. As countermeasures, we 1) introduce auxiliary tasks with quadruplet loss functions to capture cross-task fine-grained ranking information and avoid task conflicts, 2) design a calibrated distillation approach to align and distill knowledge from auxiliary tasks, and 3) propose a novel error correction mechanism to enable and facilitate synchronous training of teacher and student models. Comprehensive experiments are conducted to verify the effectiveness of our framework in real-world datasets.