# Privileged Features Distillation at Taobao Recommendations

arXiv: https://arxiv.org/abs/1907.05171 · HTML: https://mlsanity.com/paper/1907.05171/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2019.

## Built on by (most cited first, 20 of 20)

- [Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community](https://mlsanity.com/paper/2307.09751/) (2023)
- [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022)
- [Toward Understanding Privileged Features Distillation in Learning-to-Rank](https://mlsanity.com/paper/2209.08754/) (2022)
- [User Response Prediction in Online Advertising](https://mlsanity.com/paper/2101.02342/) (2021)
- [Transfer and Marginalize: Explaining Away Label Noise with Privileged Information](https://mlsanity.com/paper/2202.09244/) (2022)
- [Enhanced Multimodal Representation Learning with Cross-modal KD](https://mlsanity.com/paper/2306.07646/) (2023)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)
- [Challenges and approaches to privacy preserving post-click conversion prediction](https://mlsanity.com/paper/2201.12666/) (2022)
- [VFed-SSD: Towards Practical Vertical Federated Advertising](https://mlsanity.com/paper/2205.15987/) (2022)
- [Cross-Video Contextual Knowledge Exploration and Exploitation for Ambiguity Reduction in Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2308.12609/) (2023)
- [Optimizing E-commerce Search: Toward a Generalizable and Rank-Consistent Pre-Ranking Model](https://mlsanity.com/paper/2405.05606/) (2024)
- [CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification](https://mlsanity.com/paper/2410.03038/) (2024)
- [Rethinking Knowledge Transfer in Learning Using Privileged Information](https://mlsanity.com/paper/2408.14319/) (2024)
- [Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation](https://mlsanity.com/paper/2211.11159/) (2022)
- [Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text](https://mlsanity.com/paper/2301.10887/) (2023)
- [Hierarchical Multi-Interest Co-Network For Coarse-Grained Ranking](https://mlsanity.com/paper/2210.10547/) (2022)
- [Bridge the Gap between Past and Future: Siamese Model Optimization for Context-Aware Document Ranking](https://mlsanity.com/paper/2505.14180/) (2025)
- [When are radiology reports useful for training medical image classifiers?](https://mlsanity.com/paper/2510.24385/) (2025)
- [Not All Candidates are Created Equal: A Heterogeneity-Aware Approach to Pre-ranking in Recommender Systems](https://mlsanity.com/paper/2603.03770/) (2026)
- [LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation](https://mlsanity.com/paper/2605.29280/) (2026)

## Abstract (arXiv)

Features play an important role in the prediction tasks of e-commerce recommendations. To guarantee the consistency of off-line training and on-line serving, we usually utilize the same features that are both available. However, the consistency in turn neglects some discriminative features. For example, when estimating the conversion rate (CVR), i.e., the probability that a user would purchase the item if she clicked it, features like dwell time on the item detailed page are informative. However, CVR prediction should be conducted for on-line ranking before the click happens. Thus we cannot get such post-event features during serving. We define the features that are discriminative but only available during training as the privileged features. Inspired by the distillation techniques which bridge the gap between training and inference, in this work, we propose privileged features distillation (PFD). We train two models, i.e., a student model that is the same as the original one and a teacher model that additionally utilizes the privileged features. Knowledge distilled from the more accurate teacher is transferred to the student to improve its accuracy. During serving, only the student part is extracted and it relies on no privileged features. We conduct experiments on two fundamental prediction tasks at Taobao recommendations, i.e., click-through rate (CTR) at coarse-grained ranking and CVR at fine-grained ranking. By distilling the interacted features that are prohibited during serving for CTR and the post-event features for CVR, we achieve significant improvements over their strong baselines. During the on-line A/B tests, the click metric is improved by +5.0% in the CTR task. And the conversion metric is improved by +2.3% in the CVR task. Besides, by addressing several issues of training PFD, we obtain comparable training speed as the baselines without any distillation.