mlsanityData to 7 Oct 2026

Just Rank: Rethinking Evaluation with Word and Sentence Similarities

arXiv 2203.02679 ↗First posted 5 Mar 2022cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have become the de facto evaluation method. It leads models to overfit to such evaluations, negatively impacting embedding models' development. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. Further, we propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future benchmarking purposes.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 37 of its 87 references that are in the map. First posted on arXiv 5 Mar 2022.

Builds on

37 of 87 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
25 more
  1. 2015Skip-Thought Vectorscited by 321
  2. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  3. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  4. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  5. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  6. 2018An efficient framework for learning sentence representationscited by 197
  7. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  8. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  9. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  10. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  11. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  12. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  13. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55
  14. 2016SimVerb-3500: A Large-Scale Evaluation Set of Verb Similaritycited by 44
  15. 2019Evaluating Word Embedding Models: Methods and Experimental Resultscited by 37
  16. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
  17. 2019Continual Learning for Sentence Representations Using Conceptorscited by 22
  18. 2018Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representationscited by 20
  19. 2019Correlation Coefficients and Semantic Textual Similaritycited by 20
  20. 2021What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Studycited by 15
  21. 2018Unsupervised Post-processing of Word Vectors via Conceptor Negationcited by 10
  22. 2019Pitfalls in the Evaluation of Sentence Embeddingscited by 9
  23. 2019Efficient Sentence Embedding using Discrete Cosine Transformcited by 7
  24. 2019In Search for Linear Relations in Sentence Embedding Spacescited by 3
  25. 2018Post-Processing of Word Representations via Variance Normalization and Dynamic Embeddingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 5; 2025: 4; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  2. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 18
  3. 2024SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languagescited by 16
  4. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  5. 2024SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languagescited by 7
  6. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  7. 2022SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluationcited by 4
  8. 2022Relational Sentence Embedding for Flexible Semantic Matchingcited by 4
  9. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  10. 2023Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learningcited by 3
  11. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
  12. 2023Interpretation modeling: Social grounding of sentences by reasoning over their implicit moral judgmentscited by 2
8 more
  1. 2023JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applicationscited by 1
  2. 2023Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Modelcited by 1
  3. 2025Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generationcited by 1
  4. 2023Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Modelscited by 0
  5. 2025Set-Theoretic Compositionality of Sentence Embeddingscited by 0
  6. 2025Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluationcited by 0
  7. 2025Hard Negative Sample-Augmented DPO Post-Training for Small Language Modelscited by 0
  8. 2026Structure Retention in Embedding Spaces as a Predictor of Benchmark Performancecited by 0