# Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering

arXiv: https://arxiv.org/abs/2005.00038 · HTML: https://mlsanity.com/paper/2005.00038/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [R2-D2: A Modular Baseline for Open-Domain Question Answering](https://mlsanity.com/paper/2109.03502/) (2021)
- [Knowledge Perceived Multi-modal Pretraining in E-commerce](https://mlsanity.com/paper/2109.00895/) (2021)
- [You Only Need One Model for Open-domain Question Answering](https://mlsanity.com/paper/2112.07381/) (2021)
- [Billion-scale Pre-trained E-commerce Product Knowledge Graph Model](https://mlsanity.com/paper/2105.00388/) (2021)
- [Open-Domain Question-Answering for COVID-19 and Other Emergent Domains](https://mlsanity.com/paper/2110.06962/) (2021)
- [Pruning the Index Contents for Memory Efficient Open-Domain QA](https://mlsanity.com/paper/2102.10697/) (2021)
- [PReGAN: Answer Oriented Passage Ranking with Weakly Supervised GAN](https://mlsanity.com/paper/2207.01762/) (2022)
- [Location Aware Modular Biencoder for Tourism Question Answering](https://mlsanity.com/paper/2401.02187/) (2024)
- [PKGM: A Pre-trained Knowledge Graph Model for E-commerce Application](https://mlsanity.com/paper/2203.00964/) (2022)
- [Enhanced vectors for top-k document retrieval in Question Answering](https://mlsanity.com/paper/2210.10584/) (2022)

## Abstract (arXiv)

To extract answers from a large corpus, open-domain question answering (QA) systems usually rely on information retrieval (IR) techniques to narrow the search space. Standard inverted index methods such as TF-IDF are commonly used as thanks to their efficiency. However, their retrieval performance is limited as they simply use shallow and sparse lexical features. To break the IR bottleneck, recent studies show that stronger retrieval performance can be achieved by pretraining a effective paragraph encoder that index paragraphs into dense vectors. Once trained, the corpus can be pre-encoded into low-dimensional vectors and stored within an index structure where the retrieval can be efficiently implemented as maximum inner product search. Despite the promising results, pretraining such a dense index is expensive and often requires a very large batch size. In this work, we propose a simple and resource-efficient method to pretrain the paragraph encoder. First, instead of using heuristically created pseudo question-paragraph pairs for pretraining, we utilize an existing pretrained sequence-to-sequence model to build a strong question generator that creates high-quality pretraining data. Second, we propose a progressive pretraining algorithm to ensure the existence of effective negative samples in each batch. Across three datasets, our method outperforms an existing dense retrieval method that uses 7 times more computational resources for pretraining.