# Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval

arXiv: https://arxiv.org/abs/2105.03599 · HTML: https://mlsanity.com/paper/2105.03599/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval](https://mlsanity.com/paper/1910.10687/) (2019)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [RepBERT: Contextualized Text Embeddings for First-Stage Retrieval](https://mlsanity.com/paper/2006.15498/) (2020)
- [DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling](https://mlsanity.com/paper/2010.03099/) (2020)
- [A Dual Embedding Space Model for Document Ranking](https://mlsanity.com/paper/1602.01137/) (2016)

## Built on by (most cited first, 14 of 14)

- [Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation](https://mlsanity.com/paper/2402.18191/) (2024)
- [Multi-View Document Representation Learning for Open-Domain Dense Retrieval](https://mlsanity.com/paper/2203.08372/) (2022)
- [Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval](https://mlsanity.com/paper/2110.07524/) (2021)
- [Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval](https://mlsanity.com/paper/2504.05181/) (2025)
- [LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding](https://mlsanity.com/paper/2404.05825/) (2024)
- [Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval](https://mlsanity.com/paper/2208.04232/) (2022)
- [SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts](https://mlsanity.com/paper/2507.13105/) (2025)
- [VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction](https://mlsanity.com/paper/2112.04195/) (2021)
- [ODSum: New Benchmarks for Open Domain Multi-Document Summarization](https://mlsanity.com/paper/2309.08960/) (2023)
- [Enhancing Open-Domain Table Question Answering via Syntax- and Structure-aware Dense Retrieval](https://mlsanity.com/paper/2309.10506/) (2023)
- [A Unified Causal View of Instruction Tuning](https://mlsanity.com/paper/2402.06220/) (2024)
- [Multi-Step Semantic Reasoning in Generative Retrieval](https://mlsanity.com/paper/2603.12368/) (2026)
- [On Elastic Language Models](https://mlsanity.com/paper/2311.07204/) (2023)
- [MSRS: Evaluating Multi-Source Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.20867/) (2025)

## Abstract (arXiv)

Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than traditional sparse vector space models. To obtain high efficiency, the basic structure of these models is Bi-encoder in most cases. However, this simple structure may cause serious information loss during the encoding of documents since the queries are agnostic. To address this problem, we design a method to mimic the queries on each of the documents by an iterative clustering process and represent the documents by multiple pseudo queries (i.e., the cluster centroids). To boost the retrieval process using approximate nearest neighbor search library, we also optimize the matching function with a two-step score calculation procedure. Experimental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results.