# BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search

arXiv: https://arxiv.org/abs/2010.10442 · HTML: https://mlsanity.com/paper/2010.10442/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Prediction](https://mlsanity.com/paper/2308.11527/) (2023)
- [Learning variant product relationship and variation attributes from e-commerce website structures](https://mlsanity.com/paper/2410.02779/) (2024)
- [Robust Interaction-Based Relevance Modeling for Online e-Commerce Search](https://mlsanity.com/paper/2406.02135/) (2024)
- [SPM: Structured Pretraining and Matching Architectures for Relevance Modeling in Meituan Search](https://mlsanity.com/paper/2308.07711/) (2023)
- [TeKo: Text-Rich Graph Neural Networks with External Knowledge](https://mlsanity.com/paper/2206.07253/) (2022)
- [Towards Better Query Classification with Multi-Expert Knowledge Condensation in JD Ads Search](https://mlsanity.com/paper/2308.01098/) (2023)

## Abstract (arXiv)

Relevance has significant impact on user experience and business profit for e-commerce search platform. In this work, we propose a data-driven framework for search relevance prediction, by distilling knowledge from BERT and related multi-layer Transformer teacher models into simple feed-forward networks with large amount of unlabeled data. The distillation process produces a student model that recovers more than 97% test accuracy of teacher models on new queries, at a serving cost that's several magnitude lower (latency 150x lower than BERT-Base and 15x lower than the most efficient BERT variant, TinyBERT). The applications of temperature rescaling and teacher model stacking further boost model accuracy, without increasing the student model complexity. We present experimental results on both in-house e-commerce search relevance data as well as a public data set on sentiment analysis from the GLUE benchmark. The latter takes advantage of another related public data set of much larger scale, while disregarding its potentially noisy labels. Embedding analysis and case study on the in-house data further highlight the strength of the resulting model. By making the data processing and model training source code public, we hope the techniques presented here can help reduce energy consumption of the state of the art Transformer models and also level the playing field for small organizations lacking access to cutting edge machine learning hardwares.