mlsanityData to 7 Oct 2026

Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval

arXiv 2312.15503 ↗First posted 24 Dec 2023cs.CLcited by 77

Not on the map. Why?

Cited by 26 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are learned by auto-regression, whose working mechanism is completely different from representing whole text as one discriminative embedding. Thus, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called Llama2Vec, which performs unsupervised adaptation of LLM for its dense retrieval application. Llama2Vec consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the LLM is prompted to reconstruct the input sentence and predict the next sentence based on its text embeddings. Llama2Vec is simple, lightweight, but highly effective. It is used to adapt LLaMA-2-7B on the Wikipedia corpus. With a moderate steps of adaptation, it substantially improves the model's fine-tuned performances on a variety of dense retrieval benchmarks. Notably, it results in the new state-of-the-art performances on popular benchmarks, such as passage and document retrieval on MSMARCO, and zero-shot retrieval on BEIR. The model and source code will be made publicly available to facilitate the future research. Our model is available at https://github.com/FlagOpen/FlagEmbedding.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 24 Dec 2023.

Builds on

36 of 44 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
24 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  5. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  6. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  8. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  9. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  10. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  11. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  12. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  13. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  14. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  15. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  16. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  17. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  18. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  19. 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
  20. 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
  21. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  22. 2022SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrievalcited by 21
  23. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  24. 2022Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencodercited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 38; 2026: 13.

77 papers.

Sort:

By area:

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  2. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  3. 2024Making Text Embedders Few-Shot Learnerscited by 78
  4. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 69
  5. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  6. 2025PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Pathscited by 60
  7. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
  8. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  9. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  10. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  11. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  12. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
38 more
  1. 2024Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generationcited by 21
  2. 2024Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessmentcited by 19
  3. 2024ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrievalcited by 18
  4. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  5. 2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
  6. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15
  7. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  8. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  9. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  10. 2024Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrievalcited by 13
  11. 2025Stop-RAG: Value-Based Retrieval Control for Iterative RAGcited by 13
  12. 2025Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Methodcited by 10
  13. 2024D2LLM: Decomposed and Distilled Large Language Models for Semantic Searchcited by 9
  14. 2024MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generationcited by 8
  15. 2024rerankers: A Lightweight Python Library to Unify Ranking Methodscited by 8
  16. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  17. 2026Multi-Vector Index Compression in Any Modalitycited by 8
  18. 2025Hierarchical Lexical Graph for Enhanced Multi-Hop Retrievalcited by 7
  19. 2026Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guaranteescited by 7
  20. 2024Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generationcited by 6
  21. 2025Enhancing Lexicon-Based Text Embeddings with Large Language Modelscited by 6
  22. 2025Training LLMs to be Better Text Embedders through Bidirectional Reconstructioncited by 6
  23. 2024UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrieverscited by 5
  24. 2025Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generationcited by 5
  25. 2025MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answeringcited by 5
  26. 2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
  27. 2025Learning an Effective Premise Retrieval Model for Efficient Mathematical Formalizationcited by 4
  28. 2025Pseudo Relevance Feedback is Enough to Close the Gap Between Small and Large Dense Retrieval Modelscited by 4
  29. 2025The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corporacited by 4
  30. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 4
  31. 2025HyFedRAG: A Federated Retrieval-Augmented Generation Framework for Heterogeneous and Privacy-Sensitive Datacited by 3
  32. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representationcited by 3
  33. 2025Language Model Re-rankers are Fooled by Lexical Similaritiescited by 2
  34. 2025Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisitioncited by 2
  35. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 2
  36. 2025Enhancing Document VQA Models via Retrieval-Augmented Generationcited by 2
  37. 2025Teaching According to Students' Aptitude: Personalized Mathematics Tutoring via Persona-, Memory-, and Forgetting-Aware LLMscited by 2
  38. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2

50 most cited of 77 All 77 (JSON)