# DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling

arXiv: https://arxiv.org/abs/2010.03099 · HTML: https://mlsanity.com/paper/2010.03099/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [Multi-View Document Representation Learning for Open-Domain Dense Retrieval](https://mlsanity.com/paper/2203.08372/) (2022)
- [Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://mlsanity.com/paper/2109.13059/) (2021)
- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval](https://mlsanity.com/paper/2105.03599/) (2021)
- [Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization](https://mlsanity.com/paper/2210.12579/) (2022)
- [SDR: Efficient Neural Re-ranking using Succinct Document Representation](https://mlsanity.com/paper/2110.02065/) (2021)
- [Improving Bilingual Lexicon Induction with Cross-Encoder Reranking](https://mlsanity.com/paper/2210.16953/) (2022)
- [EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction](https://mlsanity.com/paper/2406.06839/) (2024)
- [VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction](https://mlsanity.com/paper/2112.04195/) (2021)
- [Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling](https://mlsanity.com/paper/2210.05261/) (2022)
- [Evaluation of Semantic Answer Similarity Metrics](https://mlsanity.com/paper/2206.12664/) (2022)
- [Joint Representations of Text and Knowledge Graphs for Retrieval and Evaluation](https://mlsanity.com/paper/2302.14785/) (2023)

## Abstract (arXiv)

Pre-trained models like BERT (Devlin et al., 2018) have dominated NLP / IR applications such as single sentence classification, text pair classification, and question answering. However, deploying these models in real systems is highly non-trivial due to their exorbitant computational costs. A common remedy to this is knowledge distillation (Hinton et al., 2015), leading to faster inference. However -- as we show here -- existing works are not optimized for dealing with pairs (or tuples) of texts. Consequently, they are either not scalable or demonstrate subpar performance. In this work, we propose DiPair -- a novel framework for distilling fast and accurate models on text pair tasks. Coupled with an end-to-end training strategy, DiPair is both highly scalable and offers improved quality-speed tradeoffs. Empirical studies conducted on both academic and real-world e-commerce benchmarks demonstrate the efficacy of the proposed approach with speedups of over 350x and minimal quality drop relative to the cross-attention teacher BERT model.