mlsanityData to 14 Oct 2026

Pretrained Transformers for Text Ranking: BERT and Beyond

arXiv 2010.06467 ↗First posted 13 Oct 2020cs.IR cs.CLcited by 267

Not on the map. Why?

Cited by 25 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 40; 2022: 53; 2023: 59; 2024: 57; 2025: 30; 2026: 28.

267 papers.

Sort:

By area:

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  6. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  7. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  8. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  9. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  10. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  11. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
  12. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
38 more
  1. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  2. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  3. 2022Task-aware Retrieval with Instructionscited by 107
  4. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  5. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  6. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 70
  7. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  8. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  9. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  10. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  11. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  12. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 55
  13. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 50
  14. 2023Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studycited by 49
  15. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 48
  16. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  17. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  18. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  19. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  20. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  21. 2022Neural Approaches to Conversational Information Retrievalcited by 42
  22. 2023Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Datacited by 40
  23. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  24. 2022Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Modelscited by 37
  25. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  26. 2021Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similaritycited by 35
  27. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generatorcited by 34
  28. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  29. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  30. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  31. 2023gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Samplingcited by 29
  32. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  33. 2024Scaling Laws For Dense Retrievalcited by 29
  34. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 28
  35. 2021MS MARCO: Benchmarking Ranking Models in the Large-Data Regimecited by 28
  36. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27
  37. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  38. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26

50 most cited of 267 All 267 (JSON)