mlsanityData to 14 Oct 2026

Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe

arXiv 2406.04165 ↗First posted 6 Jun 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Text embeddings are essential for many tasks, such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

35 of 45 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
23 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  12. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  13. 2023Scaling Data-Constrained Language Modelscited by 403
  14. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  15. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  16. 2023Improving Text Embeddings with Large Language Modelscited by 303
  17. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  18. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  19. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  20. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
  21. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
  22. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  23. 2022Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9
  2. 2025Enhancing Lexicon-Based Text Embeddings with Large Language Modelscited by 6
  3. 2025Contrastive Representations for Temporal Reasoningcited by 6
  4. 2025Attention with Trained Embeddings Provably Selects Important Tokenscited by 1