mlsanityData to 7 Oct 2026

Language Models are Universal Embedders

arXiv 2310.08232 ↗First posted 12 Oct 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 62 of its 85 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

62 of 85 references.

References · 62

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  10. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  11. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  13. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  14. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  15. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  16. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  17. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  18. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  19. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  20. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  21. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  22. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  23. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  24. 2019Document Expansion by Query Predictioncited by 225
  25. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  26. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  27. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  28. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  29. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  30. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  31. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  32. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  33. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  34. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  35. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  36. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  37. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  38. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 4.

11 papers.

  1. 2023Improving Text Embeddings with Large Language Modelscited by 295
  2. 2024Generative Representational Instruction Tuningcited by 227
  3. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  4. 2024Repetition Improves Language Model Embeddingscited by 70
  5. 2024Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Mergingcited by 14
  6. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  7. 2024Extracting Paragraphs from LLM Token Activationscited by 9
  8. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 8
  9. 2026Automatic Layer Selection for Hallucination Detectioncited by 3
  10. 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 2
  11. 2026Reasoning Quality Matters: Combating Reasoning Collapse in LLM-based Embedding Learningcited by 0