mlsanityData to 7 Oct 2026

Scaling Sentence Embeddings with Large Language Models

arXiv 2307.16645 ↗First posted 31 Jul 2023cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently garnered significant interest. With in-context learning, LLMs achieve impressive results in various natural language tasks. However, the application of LLMs to sentence embeddings remains an area of ongoing research. In this work, we propose an in-context learning-based method aimed at improving sentence embeddings performance. Our approach involves adapting the previous prompt-based representation method for autoregressive models, constructing a demonstration set that enables LLMs to perform in-context learning, and scaling up the LLMs to different model sizes. Through extensive experiments, in-context learning enables LLMs to generate high-quality sentence embeddings without any fine-tuning. It helps LLMs achieve performance comparable to current contrastive learning methods. By scaling model size, we find scaling to more than tens of billion parameters harms the performance on semantic textual similarity (STS) tasks. However, the largest model outperforms other counterparts and achieves the new state-of-the-art result on transfer tasks. We also fine-tune LLMs with current contrastive learning approach, and the 2.7B OPT model, incorporating our prompt-based method, surpasses the performance of 4.8B ST5, achieving the new state-of-the-art results on STS tasks. Our code is available at https://github.com/kongds/scaling_sentemb.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

32 of 53 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
20 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  13. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  14. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  15. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  16. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  17. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  18. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  19. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  20. 2021DefSent: Sentence Embeddings using Definition Sentencescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 25; 2025: 33; 2026: 22.

83 papers.

Sort:

By area:

  1. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  2. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  3. 2024Generative Representational Instruction Tuningcited by 227
  4. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  5. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  6. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  7. 2023AnglE-optimized Text Embeddingscited by 98
  8. 2024Repetition Improves Language Model Embeddingscited by 70
  9. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  10. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  11. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  12. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
38 more
  1. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  2. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  3. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  4. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  5. 2023Meaning Representations from Trajectories in Autoregressive Modelscited by 18
  6. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 18
  7. 2024Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatmentscited by 16
  8. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  9. 2024OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMscited by 14
  10. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  11. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 13
  12. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  13. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  14. 2025LLM Embeddings for Deep Learning on Tabular Datacited by 11
  15. 2025UniECS: Unified Multimodal E-Commerce Search Framework with Gated Cross-modal Fusioncited by 10
  16. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  17. 2025F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Datacited by 8
  18. 2025CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddingscited by 7
  19. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  20. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  21. 2026ELVA: Exploring Ranking-Driven Universal Multimodal Retrievalcited by 5
  22. 2024Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encodingcited by 4
  23. 2024Editing Knowledge Representation of Language Model via Rephrased Prefix Promptscited by 4
  24. 2024REAL: Response Embedding-based Alignment for LLMscited by 4
  25. 2026The Embedder's Dilemma: LLMs Are Better, but at What Cost?cited by 4
  26. 2024Linearly-Interpretable Concept Embedding Models for Text Analysiscited by 3
  27. 2024Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifierscited by 3
  28. 2025Analyzing Similarity Metrics for Data Selection for Language Model Pretrainingcited by 3
  29. 2025CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Passcited by 3
  30. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  31. 2025DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learningcited by 2
  32. 2025Histopathology Multi-modal Embedding for Pathology Composed Retrievalcited by 2
  33. 2025Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPscited by 2
  34. 2025Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrievalcited by 2
  35. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  36. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 2
  37. 2026LLM2Vec-Gen: Generative Embeddings from Large Language Modelscited by 2
  38. 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 2

50 most cited of 83 All 83 (JSON)