mlsanityData to 7 Oct 2026

Gecko: Versatile Text Embeddings Distilled from Large Language Models

arXiv 2403.20327 ↗First posted 29 Mar 2024cs.CL cs.AIcited by 81

Not on the map. Why?

Abstract (arXiv)

We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness of our approach is demonstrated by the compactness of the Gecko. On the Massive Text Embedding Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing entries with 768 embedding size. Gecko with 768 embedding dimensions achieves an average score of 66.31, competing with 7x larger models and 5x higher dimensional embeddings.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

42 of 49 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
30 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2014Distributed Representations of Sentences and Documentscited by 677
  6. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  7. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  11. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  12. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  13. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  14. 2023Improving Text Embeddings with Large Language Modelscited by 295
  15. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  16. 2022Matryoshka Representation Learningcited by 268
  17. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  18. 2024Generative Representational Instruction Tuningcited by 227
  19. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  20. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  21. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  22. 2022Task-aware Retrieval with Instructionscited by 107
  23. 2020Learning Dense Representations of Phrases at Scalecited by 91
  24. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  25. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  26. 2024Repetition Improves Language Model Embeddingscited by 70
  27. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  28. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  29. 2024Leveraging LLMs for Unsupervised Dense Retriever Rankingcited by 19
  30. 2023SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negativescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 45; 2026: 11.

81 papers.

Sort:

By area:

  1. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  2. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  3. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  4. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  5. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  6. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  7. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  8. 2024Making Text Embedders Few-Shot Learnerscited by 78
  9. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
  10. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  11. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 57
  12. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
38 more
  1. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  2. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  3. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  4. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  5. 2025DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrievalcited by 35
  6. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generationcited by 34
  7. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  8. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  9. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  10. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
  11. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  12. 2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
  13. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  14. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  15. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  16. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  17. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  18. 2024Linq-Embed-Mistral Technical Reportcited by 18
  19. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?cited by 15
  20. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  21. 2025Stop-RAG: Value-Based Retrieval Control for Iterative RAGcited by 13
  22. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  23. 2024Piccolo2: General Text Embedding with Multi-task Hybrid Loss Trainingcited by 10
  24. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  25. 2025Towards Better Instruction Following Retrieval Modelscited by 9
  26. 2024STAR: A Simple Training-free Approach for Recommendations using Large Language Modelscited by 8
  27. 2025Writing as a testbed for open ended agentscited by 8
  28. 2025F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Datacited by 8
  29. 2025Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Datacited by 7
  30. 2025Extracting alignment data in open modelscited by 7
  31. 2025Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?cited by 6
  32. 2025Generative Data Refinement: Just Ask for Better Datacited by 6
  33. 2025Learning Refined Document Representations for Dense Retrieval via Deliberate Thinkingcited by 5
  34. 2025MIRB: Mathematical Information Retrieval Benchmarkcited by 5
  35. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  36. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 5
  37. 2024Understanding Generative AI Content with Embedding Modelscited by 4
  38. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4

50 most cited of 81 All 81 (JSON)