# PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation

arXiv: https://arxiv.org/abs/2211.06059 · HTML: https://mlsanity.com/paper/2211.06059/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2022.

## Built on by (most cited first, 3 of 3)

- [DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion](https://mlsanity.com/paper/2406.06567/) (2024)
- [Best Practices for Distilling Large Language Models into BERT for Web Search Ranking](https://mlsanity.com/paper/2411.04539/) (2024)
- [GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation](https://mlsanity.com/paper/2405.03764/) (2024)

## Abstract (arXiv)

Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely used. In this paper, we focus on two key questions in knowledge distillation for ranking models: 1) how to ensemble knowledge from multi-teacher; 2) how to utilize the label information of data in the distillation process. We propose a unified algorithm called Pairwise Iterative Logits Ensemble (PILE) to tackle these two questions simultaneously. PILE ensembles multi-teacher logits supervised by label information in an iterative way and achieved competitive performance in both offline and online experiments. The proposed method has been deployed in a real-world commercial search system.