mlsanityData to 7 Oct 2026

Making Text Embedders Few-Shot Learners

arXiv 2409.15700 ↗First posted 24 Sep 2024cs.IR cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) with decoder-only architectures demonstrate remarkable in-context learning (ICL) capabilities. This feature enables them to effectively handle both familiar and novel tasks by utilizing examples provided within their input context. Recognizing the potential of this capability, we propose leveraging the ICL feature in LLMs to enhance the process of text embedding generation. To this end, we introduce a novel model bge-en-icl, which employs few-shot examples to produce high-quality text embeddings. Our approach integrates task-related examples directly into the query side, resulting in significant improvements across various tasks. Additionally, we have investigated how to effectively utilize LLMs as embedding models, including various attention mechanisms, pooling methods, etc. Our findings suggest that retaining the original framework often yields the best results, underscoring that simplicity is best. Experimental results on the MTEB and AIR-Bench benchmarks demonstrate that our approach sets new state-of-the-art (SOTA) performance. Our model, code and dataset are freely available at https://github.com/FlagOpen/FlagEmbedding .

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

49 of 68 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023Mistral 7Bcited by 3,065
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
37 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  12. 2019ELI5: Long Form Question Answeringcited by 567
  13. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  14. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  15. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  16. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  17. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  18. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  19. 2023Improving Text Embeddings with Large Language Modelscited by 295
  20. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  21. 2024Generative Representational Instruction Tuningcited by 227
  22. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  23. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  24. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  25. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  26. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  27. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  28. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  29. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  30. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  31. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 44
  32. 2024PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrievalcited by 30
  33. 2022Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrievalcited by 26
  34. 2024PL-MTEB: Polish Massive Text Embedding Benchmarkcited by 19
  35. 2024Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessmentcited by 19
  36. 2022DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Enginecited by 15
  37. 2024MTEB-French: Resources for French Sentence Embedding Evaluation and Analysiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 56; 2026: 16.

78 papers.

Sort:

By area:

  1. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  2. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  3. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  4. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  5. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  6. 2025LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendationcited by 19
  7. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  8. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  9. 2025Dynamic and Generalizable Process Reward Modelingcited by 16
  10. 2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
  11. 2025ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrievalcited by 15
  12. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
38 more
  1. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  2. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 10
  3. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  4. 2025Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinementcited by 9
  5. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  6. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  7. 2025Model Generalization on Text Attribute Graphs: Principles with Large Language Modelscited by 8
  8. 2025F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Datacited by 8
  9. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 8
  10. 2025mFollowIR: a Multilingual Benchmark for Instruction Following in Retrievalcited by 7
  11. 2024Bilingual BSARD: Extending Statutory Article Retrieval to Dutchcited by 5
  12. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  13. 2026LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculumcited by 5
  14. 2026MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for P=ali, Sanskrit, Buddhist Chinese, and Tibetancited by 5
  15. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 5
  16. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarkingcited by 4
  17. 2025Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddingscited by 4
  18. 2025TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? -- A Case Study on Korea Financial Textscited by 3
  19. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 3
  20. 2025From Chat Logs to Collective Insights: Aggregative Question Answeringcited by 3
  21. 2025LGAI-EMBEDDING-Preview Technical Reportcited by 3
  22. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  23. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  24. 2025QZhou-Embedding Technical Reportcited by 3
  25. 2025Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddingscited by 3
  26. 2025E2Rank: Unifying Text Embedding and Listwise Reranking for Effective and Efficient Searchcited by 3
  27. 2025Large Language Models Meet Extreme Multi-label Classification: Scaling and Multi-modal Frameworkcited by 3
  28. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representationcited by 3
  29. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  30. 2026Automatic Layer Selection for Hallucination Detectioncited by 3
  31. 2024RARe: Retrieval Augmented Retrieval with In-Context Examplescited by 2
  32. 2025Scale-Free Graph-Language Modelscited by 2
  33. 2025FRASE: Structured Representations for Generalizable SPARQL Query Generationcited by 2
  34. 2025CLIRudit: Cross-Lingual Information Retrieval of Scientific Documentscited by 2
  35. 2025ExpeTrans: LLMs Are Experiential Transfer Learnerscited by 2
  36. 2025SUCEA: Reasoning-Intensive Retrieval for Adversarial Fact-checking through Claim Decomposition and Editingcited by 2
  37. 2025HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrievalcited by 2
  38. 2025LLM-based Embedders for Prior Case Retrievalcited by 2

50 most cited of 78 All 78 (JSON)