mlsanityData to 14 Oct 2026

Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation

arXiv 2203.07687 ↗First posted 15 Mar 2022cs.CL cs.IRcited by 5

Not on the map. Why?

Abstract (arXiv)

How to learn highly compact yet effective sentence representation? Pre-trained language models have been effective in many NLP tasks. However, these models are often huge and produce large sentence embeddings. Moreover, there is a big performance gap between large and small models. In this paper, we propose Homomorphic Projective Distillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to produce compact representations while mimicking a large pre-trained language model to retain the sentence representation quality. We evaluate our method with different model sizes on both semantic textual similarity (STS) and semantic retrieval (SR) tasks. Experiments show that our method achieves 2.7-4.5 points performance gain on STS tasks compared with previous best representations of the same size. In SR tasks, our method improves retrieval speed (8.2×) and memory usage (8.0×) compared with state-of-the-art large models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

28 of 41 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
16 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  5. 2015Skip-Thought Vectorscited by 322
  6. 2019Hyperbolic Image Embeddingscited by 221
  7. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  8. 2018An efficient framework for learning sentence representationscited by 198
  9. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 104
  10. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
  11. 2015Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrievalcited by 54
  12. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  13. 2017Simple and Effective Dimensionality Reduction for Word Embeddingscited by 39
  14. 2018Exploiting Sentence Embedding for Medical Question Answeringcited by 5
  15. 2018Understanding and Improving Multi-Sense Word Embeddings via Extended Robust Principal Component Analysiscited by 1
  16. 2020A Multi-Semantic Metapath Model for Large Scale Heterogeneous Network Representation Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 2; 2026: 1.

5 papers.

  1. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  2. 2023"Private Prediction Strikes Back!'' Private Kernelized Nearest Neighbors with Individual Renyi Filtercited by 5
  3. 2023Text Representation Distillation via Information Bottleneck Principlecited by 3
  4. 2026On the Representational Limits of Quantum-Inspired 1024-D Document Embeddings: An Experimental Evaluation Frameworkcited by 1
  5. 2025Compressed Concatenation of Small Embedding Modelscited by 0