# ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

arXiv: https://arxiv.org/abs/2609.23231 · HTML: https://mlsanity.com/paper/2609.23231/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting](https://mlsanity.com/paper/2105.11698/) (2021)
- [Varifocal Question Generation for Fact-checking](https://mlsanity.com/paper/2210.12400/) (2022)
- [QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation](https://mlsanity.com/paper/2406.05707/) (2024)
- [Overview of the TREC 2024 NeuCLIR Track](https://mlsanity.com/paper/2509.14355/) (2025)
- [ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain](https://mlsanity.com/paper/2412.00532/) (2024)
- [Efficient Terminology Integration for LLM-based Translation in Specialized Domains](https://mlsanity.com/paper/2410.15690/) (2024)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [Bridging Language Gaps: Advances in Cross-Lingual Information Retrieval with Multilingual LLMs](https://mlsanity.com/paper/2510.00908/) (2025)
- [The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora](https://mlsanity.com/paper/2507.07543/) (2025)
- [ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings](https://mlsanity.com/paper/2508.01643/) (2025)
- [CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents](https://mlsanity.com/paper/2504.16264/) (2025)
- [What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models](https://mlsanity.com/paper/2511.19324/) (2025)

## Abstract (arXiv)

Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a different language than the query. However, existing benchmarks do not adequately capture domain-specific cross-lingual retrieval or the retrieval-depth and recoverability failures that aggregate recall hides. In this work, we benchmark CLIR in the chemical domain, with a focus on patent data. We construct a multilingual dataset from Google Patents and the European Patent Office (EPO) data, spanning five languages (covering major Eastern and Western languages) and reflecting the diversity and complexity of real-world industrial documentation. Using this dataset, we systematically evaluate eight state-of-the-art embedding models for cross-lingual retrieval. Our results show a substantial performance gap between monolingual and cross-lingual settings: for the best-performing model, Recall@10 drops from 0.72 to 0.53 in cross-lingual setting. Retrieval depth also degrades significantly, with relevant documents ranked lower across languages in cross-lingual scenarios. Furthermore, some multilingual embedding models that perform strongly in monolingual settings exhibit sharp declines when queries and documents are in different languages, providing practical insights for model selection in cross-lingual use cases. These findings highlight critical limitations of current approaches and emphasize the need for more robust cross-lingual retrieval methods in domain-specific settings. Our benchmark provides actionable insights for model selection and establishes a controlled diagnostic evaluation framework for CLIR over industrial technical text. Data and code are publicly available at https://github.com/MohammadKhodadad/Multi-Lingual-QAC.