# Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges

arXiv: https://arxiv.org/abs/2403.00784 · HTML: https://mlsanity.com/paper/2403.00784/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 86 of its 160 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 30 of 86)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)

## Built on by (most cited first, 20 of 20)

- [Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.02464/) (2025)
- [Let Community Rules Be Reflected in Online Content Moderation](https://mlsanity.com/paper/2408.12035/) (2024)
- [LREF: A Novel LLM-based Relevance Framework for E-commerce](https://mlsanity.com/paper/2503.09223/) (2025)
- [From Ranking to Selection: A Simple but Efficient Dynamic Passage Selector for Retrieval Augmented Generation](https://mlsanity.com/paper/2508.09497/) (2025)
- [TRACE: Transparent Web Reliability Assessment with Contextual Explanations](https://mlsanity.com/paper/2506.12072/) (2025)
- [DRAFT-ing Architectural Design Decisions using LLMs](https://mlsanity.com/paper/2504.08207/) (2025)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments](https://mlsanity.com/paper/2503.08379/) (2025)
- [MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training](https://mlsanity.com/paper/2502.20855/) (2025)
- [Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research](https://mlsanity.com/paper/2604.17843/) (2026)
- [Towards Zero-Shot, Controllable Dialog Planning with LLMs](https://mlsanity.com/paper/2410.05821/) (2024)
- [Multi-Objective Planning with Contextual Lexicographic Reward Preferences](https://mlsanity.com/paper/2502.10476/) (2025)
- [A Split-Window Transformer for Multi-Model Sequence Spammer Detection using Multi-Model Variational Autoencoder](https://mlsanity.com/paper/2502.16483/) (2025)
- [CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents](https://mlsanity.com/paper/2506.03822/) (2025)
- [Large model retrieval enhancement framework for construction site risk identification](https://mlsanity.com/paper/2508.02073/) (2025)
- [Cross-view Localization and Synthesis -- Datasets, Challenges and Opportunities](https://mlsanity.com/paper/2510.22736/) (2025)
- [ChiMDQA: Towards Comprehensive Chinese Document QA with Fine-grained Evaluation](https://mlsanity.com/paper/2511.03656/) (2025)
- [QUARK: Robust Retrieval under Non-Faithful Queries via Query-Anchored Aggregation](https://mlsanity.com/paper/2601.21049/) (2026)
- [AlertBERT: A noise-robust alert grouping framework for simultaneous cyber attacks](https://mlsanity.com/paper/2602.06534/) (2026)
- [PREBA: Surgical Duration Prediction via PCA-Weighted Retrieval-Augmented LLMs and Bayesian Averaging Aggregation](https://mlsanity.com/paper/2603.13275/) (2026)

## Abstract (arXiv)

Recent years have witnessed a substantial increase in the use of deep learning to solve various natural language processing (NLP) problems. Early deep learning models were constrained by their sequential or unidirectional nature, such that they struggled to capture the contextual relationships across text inputs. The introduction of bidirectional encoder representations from transformers (BERT) leads to a robust encoder for the transformer model that can understand the broader context and deliver state-of-the-art performance across various NLP tasks. This has inspired researchers and practitioners to apply BERT to practical problems, such as information retrieval (IR). A survey that focuses on a comprehensive analysis of prevalent approaches that apply pretrained transformer encoders like BERT to IR can thus be useful for academia and the industry. In light of this, we revisit a variety of BERT-based methods in this survey, cover a wide range of techniques of IR, and group them into six high-level categories: (i) handling long documents, (ii) integrating semantic information, (iii) balancing effectiveness and efficiency, (iv) predicting the weights of terms, (v) query expansion, and (vi) document expansion. We also provide links to resources, including datasets and toolkits, for BERT-based IR systems. A key highlight of our survey is the comparison between BERT's encoder-based models and the latest generative Large Language Models (LLMs), such as ChatGPT, which rely on decoders. Despite the popularity of LLMs, we find that for specific tasks, finely tuned BERT encoders still outperform, and at a lower deployment cost. Finally, we summarize the comprehensive outcomes of the survey and suggest directions for future research in the area.