mlsanityData to 7 Oct 2026

When is an Embedding Model More Promising than Another?

arXiv 2406.07640 ↗First posted 11 Jun 2024cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Embedders play a central role in machine learning, projecting any object into numerical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends on domain-specific empirical approaches utilizing downstream tasks, primarily because of the lack of a standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove to be prohibitively expensive and time-consuming. In this paper, we present a unified approach to evaluate embedders. First, we establish theoretical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading to a task-agnostic and self-supervised ranking procedure. We demonstrate experimentally that our approach aligns closely with the capability of embedding models to facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 54 of its 111 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

54 of 111 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2017Graph Attention Networkscited by 8,854
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
38 more
  1. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  4. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  7. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  8. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  9. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  11. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  12. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  13. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  14. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  15. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  16. 2018GuacaMol: Benchmarking Models for De Novo Molecular Designcited by 294
  17. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  18. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  19. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  20. 2020GEOM: Energy-annotated molecular conformations for property prediction and molecular generationcited by 238
  21. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  22. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  23. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  24. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  25. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  26. 2021Self-supervised Graph-level Representation Learning with Local and Global Structurecited by 125
  27. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  28. 2017Topology Adaptive Graph Convolutional Networkscited by 112
  29. 2014Word Representations via Gaussian Embeddingcited by 96
  30. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 92
  31. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  32. 2022Does GNN Pretraining Help Molecular Representation?cited by 59
  33. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  34. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  35. 2023Fractional Denoising for 3D Molecular Pre-trainingcited by 33
  36. 2021Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Taskscited by 32
  37. 2022Extreme Generative Image Compression by Learning Text Embedding from Diffusion Modelscited by 30
  38. 2018Evaluation of sentence embeddings in downstream and linguistic probing taskscited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Towards an Explainable Comparison and Alignment of Feature Embeddingscited by 9
  2. 2025From Topology to Retrieval: Decoding Embedding Spaces with Unified Signaturescited by 4
  3. 2026KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Modelscited by 2
  4. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  5. 2026Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMscited by 1
  6. 2026Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Designcited by 0
  7. 2026FLARE: Task-agnostic embedding model evaluation through a normalization processcited by 0