mlsanityData to 7 Oct 2026

Whitening Not Recommended for Classification Tasks in LLMs

arXiv 2407.12886 ↗First posted 16 Jul 2024cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs). However, we find that the efficacy of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. We also explored a variety of whitening operations, including PCA, ZCA, PCA-Cor, ZCA-Cor and Cholesky whitenings. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

15 of 22 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2014Distributed Representations of Sentences and Documentscited by 677
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  9. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  10. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  11. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  12. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
3 more
  1. 2018Whitening and Coloring batch transform for GANscited by 27
  2. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  3. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answerscited by 0