# Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset

arXiv: https://arxiv.org/abs/2112.01810 · HTML: https://mlsanity.com/paper/2112.01810/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [RepBERT: Contextualized Text Embeddings for First-Stage Retrieval](https://mlsanity.com/paper/2006.15498/) (2020)
- [ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search](https://mlsanity.com/paper/2006.05324/) (2020)
- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval](https://mlsanity.com/paper/2002.06275/) (2020)

## Built on by (most cited first, 7 of 7)

- [Czech Dataset for Cross-lingual Subjectivity Classification](https://mlsanity.com/paper/2204.13915/) (2022)
- [Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Model](https://mlsanity.com/paper/2307.14785/) (2023)
- [Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks](https://mlsanity.com/paper/2508.08125/) (2025)
- [Some Like It Small: Czech Semantic Embedding Models for Industry Applications](https://mlsanity.com/paper/2311.13921/) (2023)
- [CWRCzech: 100M Query-Document Czech Click Dataset and Its Application to Web Relevance Ranking](https://mlsanity.com/paper/2405.20994/) (2024)
- [A Comparative Study of Text Retrieval Models on DaReCzech](https://mlsanity.com/paper/2411.12921/) (2024)
- [CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents](https://mlsanity.com/paper/2603.03884/) (2026)

## Abstract (arXiv)

Web search engines focus on serving highly relevant results within hundreds of milliseconds. Pre-trained language transformer models such as BERT are therefore hard to use in this scenario due to their high computational demands. We present our real-time approach to the document ranking problem leveraging a BERT-based siamese architecture. The model is already deployed in a commercial search engine and it improves production performance by more than 3%. For further research and evaluation, we release DaReCzech, a unique data set of 1.6 million Czech user query-document pairs with manually assigned relevance levels. We also release Small-E-Czech, an Electra-small language model pre-trained on a large Czech corpus. We believe this data will support endeavours both of search relevance and multilingual-focused research communities.