# BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language

arXiv: https://arxiv.org/abs/2305.19840 · HTML: https://mlsanity.com/paper/2305.19840/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval](https://mlsanity.com/paper/2108.08787/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://mlsanity.com/paper/2105.01735/) (2021)
- [LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval](https://mlsanity.com/paper/2203.06169/) (2022)
- [Pre-training Polish Transformer-based Language Models at Scale](https://mlsanity.com/paper/2006.04229/) (2020)
- [Evaluation of Transfer Learning for Polish with a Text-to-Text Model](https://mlsanity.com/paper/2205.08808/) (2022)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [MAUPQA: Massive Automatically-created Polish Question Answering Dataset](https://mlsanity.com/paper/2305.05486/) (2023)

## Built on by (most cited first, 15 of 15)

- [PL-MTEB: Polish Massive Text Embedding Benchmark](https://mlsanity.com/paper/2405.10138/) (2024)
- [PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods](https://mlsanity.com/paper/2402.13350/) (2024)
- [BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language](https://mlsanity.com/paper/2412.08329/) (2024)
- [Silver Retriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://mlsanity.com/paper/2309.08469/) (2023)
- [Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval](https://mlsanity.com/paper/2311.01870/) (2023)
- [VN-MTEB: Vietnamese Massive Text Embedding Benchmark](https://mlsanity.com/paper/2507.21500/) (2025)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [Assessing generalization capability of text ranking models in Polish](https://mlsanity.com/paper/2402.14318/) (2024)
- [Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5](https://mlsanity.com/paper/2409.05401/) (2024)
- [Building Russian Benchmark for Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2504.12879/) (2025)
- [Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction](https://mlsanity.com/paper/2408.02337/) (2024)
- [Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoders](https://mlsanity.com/paper/2410.04620/) (2024)
- [HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions](https://mlsanity.com/paper/2606.22778/) (2026)
- [MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese](https://mlsanity.com/paper/2607.04581/) (2026)
- [Polish ModernBERT: The Long and Short of Polish Language Understanding](https://mlsanity.com/paper/2609.01379/) (2026)

## Abstract (arXiv)

The BEIR dataset is a large, heterogeneous benchmark for Information Retrieval (IR) in zero-shot settings, garnering considerable attention within the research community. However, BEIR and analogous datasets are predominantly restricted to the English language. Our objective is to establish extensive large-scale resources for IR in the Polish language, thereby advancing the research in this NLP area. In this work, inspired by mMARCO and Mr.~TyDi datasets, we translated all accessible open IR datasets into Polish, and we introduced the BEIR-PL benchmark -- a new benchmark which comprises 13 datasets, facilitating further development, training and evaluation of modern Polish language models for IR tasks. We executed an evaluation and comparison of numerous IR models on the newly introduced BEIR-PL benchmark. Furthermore, we publish pre-trained open IR models for Polish language,d marking a pioneering development in this field. Additionally, the evaluation revealed that BM25 achieved significantly lower scores for Polish than for English, which can be attributed to high inflection and intricate morphological structure of the Polish language. Finally, we trained various re-ranking models to enhance the BM25 retrieval, and we compared their performance to identify their unique characteristic features. To ensure accurate model comparisons, it is necessary to scrutinise individual results rather than to average across the entire benchmark. Thus, we thoroughly analysed the outcomes of IR models in relation to each individual data subset encompassed by the BEIR benchmark. The benchmark data is available at URL bf https://huggingface.co/clarin-knext.