mlsanityData to 7 Oct 2026

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

arXiv 2405.17428 ↗First posted 27 May 2024cs.CL cs.AI cs.IRcited by 466

Not on the map. Why?

Cited by 88 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Decoder-only LLM-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce NV-Embed, incorporating architectural designs, training procedures, and curated datasets to significantly enhance the performance of LLM as a versatile embedding model, while maintaining its simplicity and reproducibility. For model architecture, we propose a latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling or using the last <EOS> token embedding from LLMs. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For training algorithm, we introduce a two-stage contrastive instruction-tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. For training data, we utilize the hard-negative mining, synthetic data generation and existing public available datasets to boost the performance of embedding model. By combining these techniques, our NV-Embed-v1 and NV-Embed-v2 models obtained the No.1 position on the MTEB leaderboard (as of May 24 and August 30, 2024, respectively) across 56 tasks, demonstrating the sustained effectiveness of the proposed methods over time. It also achieved the highest scores in the Long Doc section and the second-highest scores in the QA section of the AIR Benchmark, which covers a range of out-of-domain information retrieval topics beyond those in MTEB. We further provide the analysis of model compression techniques for generalist embedding models.

Key facts. Cited by 466 papers in the map (data through 7 Oct 2026). Builds on 51 of its 81 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

51 of 81 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2023Mistral 7Bcited by 3,065
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  13. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  14. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  15. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  16. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  17. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  18. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  19. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  20. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  21. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  22. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  23. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  24. 2023Improving Text Embeddings with Large Language Modelscited by 295
  25. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  26. 2024Generative Representational Instruction Tuningcited by 227
  27. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 199
  28. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  29. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  30. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  31. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  32. 2022Task-aware Retrieval with Instructionscited by 107
  33. 2023AnglE-optimized Text Embeddingscited by 98
  34. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  35. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  36. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  37. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  38. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 44

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 60; 2025: 238; 2026: 168.

466 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  3. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  4. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  5. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  6. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  7. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  8. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  9. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  10. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  11. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  12. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
38 more
  1. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  2. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  3. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  4. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  5. 2024Making Text Embedders Few-Shot Learnerscited by 78
  6. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  7. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 64
  8. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  9. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  10. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 57
  11. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  12. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  13. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  14. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  15. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  16. 2025Retrieval-Augmented Generation with Hierarchical Knowledgecited by 46
  17. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  18. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  19. 2025GFM-RAG: Graph Foundation Model for Retrieval Augmented Generationcited by 40
  20. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  21. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  22. 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 37
  23. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  24. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  25. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  26. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  27. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  28. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  29. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  30. 2024Towards Foundation Models for Mixed Integer Linear Programmingcited by 30
  31. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  32. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  33. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  34. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  35. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  36. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  37. 2025REARANK: Reasoning Re-ranking Agent via Reinforcement Learningcited by 24
  38. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23

50 most cited of 466 All 466 (JSON)