mlsanityData to 7 Oct 2026

Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval

arXiv 2308.08285 ↗First posted 16 Aug 2023cs.IR cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we systematically study the potential of pre-training with Large Language Model(LLM)-based document expansion for dense passage retrieval. Concretely, we leverage the capabilities of LLMs for document expansion, i.e. query generation, and effectively transfer expanded knowledge to retrievers using pre-training strategies tailored for passage retrieval. These strategies include contrastive learning and bottlenecked query generation. Furthermore, we incorporate a curriculum learning strategy to reduce the reliance on LLM inferences. Experimental results demonstrate that pre-training with LLM-based document expansion significantly boosts the retrieval performance on large-scale web-search tasks. Our work shows strong zero-shot and out-of-domain retrieval abilities, making it more widely applicable for retrieval when initializing with no human-labeled data.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

36 of 49 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
24 more
  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  3. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  4. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  5. 2020Overview of the TREC 2019 deep learning trackcited by 282
  6. 2019Document Expansion by Query Predictioncited by 225
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021Overview of the TREC 2020 deep learning trackcited by 193
  9. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  10. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  11. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  12. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  13. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  14. 2023Query Expansion by Prompting Large Language Modelscited by 56
  15. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  16. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  17. 2021Pre-trained Language Model for Web-scale Retrieval in Baidu Searchcited by 31
  18. 2019FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevancecited by 24
  19. 2022Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Predictioncited by 23
  20. 2022ConTextual Masked Auto-Encoder for Dense Passage Retrievalcited by 16
  21. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  22. 2022Query-as-context Pre-training for Dense Passage Retrievalcited by 9
  23. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  24. 2023CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrievalcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 3.

7 papers.

  1. 2024Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrievalcited by 50
  2. 2024RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Enginecited by 6
  3. 2024Self-Retrieval: End-to-End Information Retrieval with One Large Language Modelcited by 4
  4. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  5. 2025LLM-VPRF: Large Language Model Based Vector Pseudo Relevance Feedbackcited by 2
  6. 2023Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agentscited by 1
  7. 2025A Representation Sharpening Framework for Zero Shot Dense Retrievalcited by 1