# RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

arXiv: https://arxiv.org/abs/2006.15498 · HTML: https://mlsanity.com/paper/2006.15498/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 30 of 46)

- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Adversarial Retriever-Ranker for dense text retrieval](https://mlsanity.com/paper/2110.03611/) (2021)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset](https://mlsanity.com/paper/2310.17609/) (2023)
- [A Replication Study of Dense Passage Retriever](https://mlsanity.com/paper/2104.05740/) (2021)
- [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022)
- [MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion](https://mlsanity.com/paper/2310.19056/) (2023)
- [CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos](https://mlsanity.com/paper/2204.00716/) (2022)
- [SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval](https://mlsanity.com/paper/2210.11773/) (2022)
- [Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges](https://mlsanity.com/paper/2403.00784/) (2024)
- [Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models](https://mlsanity.com/paper/2406.14848/) (2024)
- [Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence](https://mlsanity.com/paper/2105.12936/) (2021)
- [Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA](https://mlsanity.com/paper/2210.05197/) (2022)
- [ConTextual Masked Auto-Encoder for Dense Passage Retrieval](https://mlsanity.com/paper/2208.07670/) (2022)
- [THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval](https://mlsanity.com/paper/2305.06812/) (2023)
- [Constructing Tree-based Index for Efficient and Effective Dense Retrieval](https://mlsanity.com/paper/2304.11943/) (2023)
- [Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval](https://mlsanity.com/paper/2105.03599/) (2021)
- [Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking](https://mlsanity.com/paper/2204.11673/) (2022)
- [Entity Retrieval for Answering Entity-Centric Questions](https://mlsanity.com/paper/2408.02795/) (2024)
- [Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval](https://mlsanity.com/paper/2504.05181/) (2025)
- [Isotropic Representation Can Improve Dense Retrieval](https://mlsanity.com/paper/2209.00218/) (2022)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)
- [Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset](https://mlsanity.com/paper/2112.01810/) (2021)
- [MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers](https://mlsanity.com/paper/2212.07841/) (2022)
- [Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models](https://mlsanity.com/paper/2503.20576/) (2025)
- [Modality-Balanced Embedding for Video Retrieval](https://mlsanity.com/paper/2204.08182/) (2022)
- [SeDR: Segment Representation Learning for Long Documents Dense Retrieval](https://mlsanity.com/paper/2211.10841/) (2022)
- [Typo-Robust Representation Learning for Dense Retrieval](https://mlsanity.com/paper/2306.10348/) (2023)

## Abstract (arXiv)

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.