mlsanityData to 7 Oct 2026

Whitening Sentence Representations for Better Semantics and Faster Retrieval

arXiv 2103.15316 ↗First posted 29 Mar 2021cs.CL cs.AI cs.LGcited by 198

Not on the map. Why?

Abstract (arXiv)

Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. Previous work has shown that the anisotropy problem is an critical bottleneck for BERT-based sentence representation which hinders the model to fully utilize the underlying semantic features. Therefore, some attempts of boosting the isotropy of sentence distribution, such as flow-based model, have been applied to sentence representations and achieved some improvement. In this paper, we find that the whitening operation in traditional machine learning can similarly enhance the isotropy of sentence representations and achieve competitive results. Furthermore, the whitening technique is also capable of reducing the dimensionality of the sentence representation. Our experimental results show that it can not only achieve promising performance but also significantly reduce the storage cost and accelerate the model retrieval speed.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

16 of 33 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2018Universal Sentence Encodercited by 485
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
4 more
  1. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  2. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  3. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  4. 2018Learning Semantic Textual Similarity from Conversationscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 53; 2023: 51; 2024: 30; 2025: 22; 2026: 28.

198 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  5. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  6. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  7. 2022Towards Universal Sequence Representation Learning for Recommender Systemscited by 120
  8. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  9. 2023AnglE-optimized Text Embeddingscited by 98
  10. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  11. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  12. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
38 more
  1. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  2. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  3. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  4. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  5. 2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
  6. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  7. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  8. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  9. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  10. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  11. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  12. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  13. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  14. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  15. 2022Pre-trained Language Models Can be Fully Zero-Shot Learnerscited by 22
  16. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  17. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  18. 2021DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddingscited by 19
  19. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 19
  20. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  21. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  22. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  23. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  24. 2022Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervisioncited by 18
  25. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  26. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  27. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  28. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  29. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  30. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  31. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  32. 2024Unlocking the Power of Large Language Models for Entity Alignmentcited by 15
  33. 2024Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learningcited by 15
  34. 2022Addressing Token Uniformity in Transformers via Singular Value Transformationcited by 14
  35. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  36. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  37. 2021Sentence Similarity Based on Contextscited by 12
  38. 2022Sentence Representation Learning with Generative Objective rather than Contrastive Objectivecited by 12

50 most cited of 198 All 198 (JSON)