mlsanityData to 14 Oct 2026

Extreme compression of sentence-transformer ranker models: faster inference, longer battery life, and less storage on edge devices

arXiv 2207.12852 ↗First posted 29 Jun 2022cs.LG cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern search systems use several large ranker models with transformer architectures. These models require large computational resources and are not suitable for usage on devices with limited computational resources. Knowledge distillation is a popular compression technique that can reduce the resource needs of such models, where a large teacher model transfers knowledge to a small student model. To drastically reduce memory requirements and energy consumption, we propose two extensions for a popular sentence-transformer distillation procedure: generation of an optimal size vocabulary and dimensionality reduction of the embedding dimension of teachers prior to distillation. We evaluate these extensions on two different types of ranker models. This results in extremely compressed student models whose analysis on a test dataset shows the significance and utility of our proposed extensions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

20 of 23 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
8 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2019How multilingual is Multilingual BERT?cited by 645
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  5. 2018Universal Sentence Encodercited by 489
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2019Extremely Small BERT Models from Mixed-Vocabulary Trainingcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  2. 2023Efficient Transformer Knowledge Distillation: A Performance Review
  3. 2021Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Survey
  5. 2023Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers
  6. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMs
  7. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
  8. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees