mlsanityData to 14 Oct 2026

SGPT: GPT Sentence Embeddings for Semantic Search

arXiv 2202.08904 ↗First posted 17 Feb 2022cs.CL cs.AI cs.IRcited by 204

Not on the map. Why?

Abstract (arXiv)

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

38 of 55 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
26 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  10. 2018Universal Sentence Encodercited by 489
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  12. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  13. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  14. 2021True Few-Shot Learning with Language Modelscited by 344
  15. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  16. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  17. 2019Multi-Stage Document Ranking with BERTcited by 201
  18. 2021Generating Datasets with Pretrained Language Modelscited by 193
  19. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  20. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  21. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
  22. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
  23. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  24. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 62
  25. 2019Classification and Clustering of Arguments with Contextualized Word Embeddingscited by 57
  26. 2020A Survey on Contextual Embeddingscited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 15; 2023: 41; 2024: 64; 2025: 57; 2026: 27.

204 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  5. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  6. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  7. 2023Scaling Data-Constrained Language Modelscited by 403
  8. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  10. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  11. 2023Improving Text Embeddings with Large Language Modelscited by 303
  12. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
38 more
  1. 2024Generative Representational Instruction Tuningcited by 233
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  3. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  5. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
  6. 2022Task-aware Retrieval with Instructionscited by 107
  7. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  8. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  9. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  10. 2024Repetition Improves Language Model Embeddingscited by 72
  11. 2024DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Modelscited by 63
  12. 2024AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Frameworkcited by 47
  13. 2023Quality-Diversity through AI Feedbackcited by 40
  14. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  15. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  16. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 36
  17. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  18. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  19. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  20. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  21. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 31
  22. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 31
  23. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  24. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  25. 2022An Efficiency Study for SPLADE Modelscited by 26
  26. 2023Large Language Models are Strong Zero-Shot Retrievercited by 26
  27. 2023When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasetscited by 26
  28. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  29. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  30. 2023CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQLcited by 24
  31. 2023MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansioncited by 23
  32. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relationscited by 23
  33. 2023Xpert: Empowering Incident Management with Query Recommendations via Large Language Modelscited by 22
  34. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  35. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  36. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  37. 2023Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankerscited by 19
  38. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19

50 most cited of 204 All 204 (JSON)