mlsanityData to 14 Oct 2026

Sentence Embeddings by Ensemble Distillation

arXiv 2104.06719 ↗First posted 14 Apr 2021cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper contributes a new State Of The Art (SOTA) for Semantic Textual Similarity (STS). We compare and combine a number of recently proposed sentence embedding methods for STS, and propose a novel and simple ensemble knowledge distillation scheme that improves on previous approaches. Our experiments demonstrate that a model trained to learn the average embedding space from multiple ensemble students outperforms all the other individual models with high robustness. Utilizing our distillation method in combination with previous methods, we significantly improve on the SOTA unsupervised STS, and by proper hyperparameter tuning of previous methods we improve the supervised SOTA scores.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

17 of 32 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  7. 2014Distributed Representations of Sentences and Documentscited by 678
  8. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  10. 2015Skip-Thought Vectorscited by 322
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2018An efficient framework for learning sentence representationscited by 198
5 more
  1. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  2. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 60
  3. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 32
  4. 2019Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embeddingcited by 12
  5. 2018Sentence Embeddings in NLI with Iterative Refinement Encoderscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Uncertainty-driven Embedding Convolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings
  2. 2024Extracting Sentence Embeddings from Pretrained Transformer Models
  3. 2024Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss
  4. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning
  5. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
  6. 2023Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
  7. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillation
  8. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddings