# Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval

arXiv: https://arxiv.org/abs/2308.08285 · HTML: https://mlsanity.com/paper/2308.08285/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [Query2doc: Query Expansion with Large Language Models](https://mlsanity.com/paper/2303.07678/) (2023)
- [SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval](https://mlsanity.com/paper/2207.02578/) (2022)
- [Query Expansion by Prompting Large Language Models](https://mlsanity.com/paper/2305.03653/) (2023)
- [Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval](https://mlsanity.com/paper/2203.05765/) (2022)
- [ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval](https://mlsanity.com/paper/2205.09153/) (2022)
- [Pre-trained Language Model for Web-scale Retrieval in Baidu Search](https://mlsanity.com/paper/2106.03373/) (2021)
- [FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance](https://mlsanity.com/paper/1905.02851/) (2019)

## Built on by (most cited first, 7 of 7)

- [Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval](https://mlsanity.com/paper/2408.01875/) (2024)
- [RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine](https://mlsanity.com/paper/2410.18097/) (2024)
- [Self-Retrieval: End-to-End Information Retrieval with One Large Language Model](https://mlsanity.com/paper/2403.00801/) (2024)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [LLM-VPRF: Large Language Model Based Vector Pseudo Relevance Feedback](https://mlsanity.com/paper/2504.01448/) (2025)
- [Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents](https://mlsanity.com/paper/2311.09513/) (2023)
- [A Representation Sharpening Framework for Zero Shot Dense Retrieval](https://mlsanity.com/paper/2511.05684/) (2025)

## Abstract (arXiv)

In this paper, we systematically study the potential of pre-training with Large Language Model(LLM)-based document expansion for dense passage retrieval. Concretely, we leverage the capabilities of LLMs for document expansion, i.e. query generation, and effectively transfer expanded knowledge to retrievers using pre-training strategies tailored for passage retrieval. These strategies include contrastive learning and bottlenecked query generation. Furthermore, we incorporate a curriculum learning strategy to reduce the reliance on LLM inferences. Experimental results demonstrate that pre-training with LLM-based document expansion significantly boosts the retrieval performance on large-scale web-search tasks. Our work shows strong zero-shot and out-of-domain retrieval abilities, making it more widely applicable for retrieval when initializing with no human-labeled data.