mlsanityData to 7 Oct 2026

WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

arXiv 2104.01767 ↗First posted 5 Apr 2021cs.CLcited by 68

Not on the map. Why?

Abstract (arXiv)

Producing the embedding of a sentence in an unsupervised way is valuable to natural language matching and retrieval problems in practice. In this work, we conduct a thorough examination of pretrained model based unsupervised sentence embeddings. We study on four pretrained models and conduct massive experiments on seven datasets regarding sentence semantics. We have there main findings. First, averaging all tokens is better than only using [CLS] vector. Second, combining both top andbottom layers is better than only using top layers. Lastly, an easy whitening-based vector normalization strategy with less than 10 lines of code consistently boosts the performance.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

28 of 44 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
16 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  4. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  7. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  8. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  9. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  10. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  11. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  12. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  13. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  14. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55
  15. 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
  16. 2018Parameter-free Sentence Embedding via Orthogonal Basiscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 17; 2023: 13; 2024: 12; 2025: 11; 2026: 12.

68 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 120
  3. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  4. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  5. 2021Learning Kernel-Smoothed Machine Translation with Retrieved Examplescited by 25
  6. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  7. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  8. 2022Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervisioncited by 18
  9. 2022Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQAcited by 17
  10. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  11. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  12. 2022Addressing Token Uniformity in Transformers via Singular Value Transformationcited by 14
38 more
  1. 2025Semantics at an Angle: When Cosine Similarity Works Until It Doesn'tcited by 13
  2. 2021Sentence Similarity Based on Contextscited by 12
  3. 2022Happenstance: Utilizing Semantic Search to Track Russian State Media Narratives about the Russo-Ukrainian War On Redditcited by 11
  4. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  5. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  6. 2024Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attackscited by 10
  7. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 10
  8. 2024Word-specific tonal realizations in Mandarincited by 9
  9. 2023Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Modelscited by 8
  10. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  11. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  12. 2022Incorporating Commonsense Knowledge into Story Ending Generation via Heterogeneous Graph Networkscited by 6
  13. 2022Holistic Sentence Embeddings for Better Out-of-Distribution Detectioncited by 6
  14. 2023UniMAP: Universal SMILES-Graph Representation Learningcited by 6
  15. 2024NNOSE: Nearest Neighbor Occupational Skill Extractioncited by 5
  16. 2022Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representationcited by 4
  17. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  18. 2025SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstractscited by 4
  19. 2023PyTrial: Machine Learning Software and Benchmark for Clinical Trial Applicationscited by 3
  20. 2024Integrating Multi-view Analysis: Multi-view Mixture-of-Expert for Textual Personality Detectioncited by 3
  21. 2024Zipfian Whiteningcited by 3
  22. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  23. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  24. 2025Digital Voices of Survival: From Social Media Disclosures to Support Provisions for Domestic Violence Victimscited by 3
  25. 2024Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similaritycited by 2
  26. 2025Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removalcited by 2
  27. 2025Demystifying Network Foundation Modelscited by 2
  28. 2026From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMscited by 2
  29. 2022Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledgecited by 1
  30. 2023Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Modelcited by 1
  31. 2023Why "classic" Transformers are shallow and how to make them go deepcited by 1
  32. 2024Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learningcited by 1
  33. 2024Whitening Not Recommended for Classification Tasks in LLMscited by 1
  34. 2025Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEBcited by 1
  35. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?cited by 1
  36. 2026The Curse and Blessing of Mean Bias in FP4-Quantized LLM Trainingcited by 1
  37. 2026Spectral Tempering for Embedding Compression in Dense Passage Retrievalcited by 1
  38. 2022Large-Scale Product Retrieval with Weakly Supervised Representation Learningcited by 0

50 most cited of 68 All 68 (JSON)