mlsanityData to 14 Oct 2026

Contrastive Learning in Distilled Models

arXiv 2401.12472 ↗First posted 23 Jan 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing models like BERT can provide state-of-the-art word embeddings for downstream NLP tasks. However, these models yet to perform well on Semantic Textual Similarity, and may be too large to be deployed as lightweight edge applications. We seek to apply a suitable contrastive learning method based on the SimCSE paper, to a model architecture adapted from a knowledge distillation based model, DistilBERT, to address these two issues. Our final lightweight model DistilFace achieves an average of 72.1 in Spearman's correlation on STS tasks, a 34.2 percent improvement over BERT base.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

11 of 20 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  11. 2021Probabilistic Contrastive Loss for Self-Supervised Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings
  2. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
  3. 2025Distillation versus Contrastive Learning: How to Train Your Rerankers
  4. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
  5. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  6. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
  7. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teachers
  8. 2024On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models