mlsanityData to 7 Oct 2026

All-but-the-Top: Simple and Effective Postprocessing for Word Representations

arXiv 1702.01417 ↗First posted 5 Feb 2017cs.CL stat.MLcited by 205

Not on the map. Why?

Abstract (arXiv)

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a em very simple, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations em even stronger. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classification) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

Key facts. Cited by 205 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 26; 2023: 22; 2024: 22; 2025: 24; 2026: 78.

205 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  3. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  4. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  5. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  6. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  7. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 64
  8. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  9. 2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
  10. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  11. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  12. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
38 more
  1. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  2. 2023Using Sequences of Life-events to Predict Human Livescited by 27
  3. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  4. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  5. 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
  6. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  7. 2023Text Rendering Strategies for Pixel Language Modelscited by 22
  8. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  9. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  10. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  11. 2023Byte Pair Encoding for Symbolic Musiccited by 19
  12. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  13. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  14. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  15. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  16. 2021A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representationscited by 16
  17. 2024How and where does CLIP process negation?cited by 16
  18. 2021Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionalitycited by 15
  19. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  20. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  21. 2022Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representationscited by 14
  22. 2022Addressing Token Uniformity in Transformers via Singular Value Transformationcited by 14
  23. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  24. 2021Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representationscited by 13
  25. 2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
  26. 2025Semantics at an Angle: When Cosine Similarity Works Until It Doesn'tcited by 13
  27. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  28. 2023Stable Anisotropic Regularizationcited by 12
  29. 2024Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approachcited by 12
  30. 2022VAST: The Valence-Assessing Semantics Test for Contextualizing Language Modelscited by 10
  31. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 10
  32. 2023Token Imbalance Adaptation for Radiology Report Generationcited by 9
  33. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
  34. 2024Dimension Reduction with Locally Adjusted Graphscited by 9
  35. 2025Metis: Training LLMs with FP4 Quantizationcited by 9
  36. 2022Isotropic Representation Can Improve Dense Retrievalcited by 8
  37. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  38. 2023Representation Of Lexical Stylistic Features In Language Models' Embedding Spacecited by 8

50 most cited of 205 All 205 (JSON)