# Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIR

arXiv: https://arxiv.org/abs/2511.05079 · HTML: https://mlsanity.com/paper/2511.05079/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 7 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis](https://mlsanity.com/paper/1912.09723/) (2019)
- [A Family of Pretrained Transformer Language Models for Russian](https://mlsanity.com/paper/2309.10931/) (2023)
- [WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia](https://mlsanity.com/paper/1608.03542/) (2016)
- [Morphological Analyzer and Generator for Russian and Ukrainian Languages](https://mlsanity.com/paper/1503.07283/) (2015)
- [The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design](https://mlsanity.com/paper/2408.12503/) (2024)
- [Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval](https://mlsanity.com/paper/2403.18405/) (2024)
- [Fun Facts: Automatic Trivia Fact Extraction from Wikipedia](https://mlsanity.com/paper/1612.03896/) (2016)
- [Building Russian Benchmark for Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2504.12879/) (2025)

## Abstract (arXiv)

In this paper, we present a novel series of Russian information retrieval datasets constructed from the "Did you know..." section of Russian Wikipedia. Our datasets support a range of retrieval tasks, including fact-checking, retrieval-augmented generation, and full-document retrieval, by leveraging interesting facts and their referenced Wikipedia articles annotated at the sentence level with graded relevance. We describe the methodology for dataset creation that enables the expansion of existing Russian Information Retrieval (IR) resources. Through extensive experiments, we extend the RusBEIR research by comparing lexical retrieval models, such as BM25, with state-of-the-art neural architectures fine-tuned for Russian, as well as multilingual models. Results of our experiments show that lexical methods tend to outperform neural models on full-document retrieval, while neural approaches better capture lexical semantics in shorter texts, such as in fact-checking or fine-grained retrieval. Using our newly created datasets, we also analyze the impact of document length on retrieval performance and demonstrate that combining retrieval with neural reranking consistently improves results. Our contribution expands the resources available for Russian information retrieval research and highlights the importance of accurate evaluation of retrieval models to achieve optimal performance. All datasets are publicly available at HuggingFace. To facilitate reproducibility and future research, we also release the full implementation on GitHub.