# Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context

arXiv: https://arxiv.org/abs/2104.08523 · HTML: https://mlsanity.com/paper/2104.08523/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2021.

## Built on by (most cited first, 6 of 6)

- [Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback](https://mlsanity.com/paper/2108.13454/) (2021)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference](https://mlsanity.com/paper/2204.11458/) (2022)
- [Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings](https://mlsanity.com/paper/2204.02710/) (2022)
- [Dialogue-Contextualized Re-ranking for Medical History-Taking](https://mlsanity.com/paper/2304.01974/) (2023)
- [Empowering Language Model with Guided Knowledge Fusion for Biomedical Document Re-ranking](https://mlsanity.com/paper/2305.04344/) (2023)

## Abstract (arXiv)

BERT-based text ranking models have dramatically advanced the state-of-the-art in ad-hoc retrieval, wherein most models tend to consider individual query-document pairs independently. In the mean time, the importance and usefulness to consider the cross-documents interactions and the query-specific characteristics in a ranking model have been repeatedly confirmed, mostly in the context of learning to rank. The BERT-based ranking model, however, has not been able to fully incorporate these two types of ranking context, thereby ignoring the inter-document relationships from the ranking and the differences among queries. To mitigate this gap, in this work, an end-to-end transformer-based ranking model, named Co-BERT, has been proposed to exploit several BERT architectures to calibrate the query-document representations using pseudo relevance feedback before modeling the relevance of a group of documents jointly. Extensive experiments on two standard test collections confirm the effectiveness of the proposed model in improving the performance of text re-ranking over strong fine-tuned BERT-Base baselines. We plan to make our implementation open source to enable further comparisons.