mlsanityData to 7 Oct 2026

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

arXiv 2104.08663 ↗First posted 17 Apr 2021cs.IR cs.AI cs.CLcited by 1,225

Not on the map. Why?

Cited by 256 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.

Key facts. Cited by 1,225 papers in the map (data through 7 Oct 2026). Builds on 50 of its 102 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

50 of 102 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2019Passage Re-ranking with BERTcited by 629
  9. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  10. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  11. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  12. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  13. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  14. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  17. 2019Document Expansion by Query Predictioncited by 225
  18. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  19. 2019Multi-Stage Document Ranking with BERTcited by 200
  20. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  21. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  22. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  23. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  24. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  25. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  26. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  27. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  28. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  29. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  30. 2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
  31. 2018End-to-End Retrieval in Continuous Spacecited by 46
  32. 2018Adversarial Domain Adaptation for Duplicate Question Detectioncited by 30
  33. 2020Complementing Lexical Retrieval with Semantic Residual Embeddingcited by 27
  34. 2020Embedding-based Zero-shot Retrieval through Query Generationcited by 24
  35. 2020MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Modelscited by 23
  36. 2020TREC-COVID: Constructing a Pandemic Information Retrieval Test Collectioncited by 23
  37. 2020MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scalecited by 20
  38. 2019ReQA: An Evaluation for End-to-End Answer Retrieval Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 90; 2023: 153; 2024: 252; 2025: 319; 2026: 383.

1,225 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  8. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  9. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  10. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  11. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
38 more
  1. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  2. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  3. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 372
  4. 2024The Faiss librarycited by 370
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  6. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  7. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  9. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  10. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  11. 2023Improving Text Embeddings with Large Language Modelscited by 295
  12. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  13. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  14. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  15. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  16. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  17. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  18. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  19. 2024Generative Representational Instruction Tuningcited by 227
  20. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  21. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  22. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  23. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  24. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  25. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  26. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  27. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  28. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  29. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  30. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  31. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  32. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  33. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  34. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  35. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  36. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  37. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  38. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123

50 most cited of 1,225 All 1,225 (JSON)