# Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages

arXiv: https://arxiv.org/abs/2412.12806 · HTML: https://mlsanity.com/paper/2412.12806/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 59 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 7 of 7)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models](https://mlsanity.com/paper/2312.02969/) (2023)
- [MessIRve: A Large-Scale Spanish Information Retrieval Dataset](https://mlsanity.com/paper/2409.05994/) (2024)
- [How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP](https://mlsanity.com/paper/2411.05527/) (2024)

## Built on by (most cited first, 8 of 8)

- [ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2502.21228/) (2025)
- [Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings](https://mlsanity.com/paper/2509.12892/) (2025)
- [Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora](https://mlsanity.com/paper/2509.17855/) (2025)
- [CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents](https://mlsanity.com/paper/2504.16264/) (2025)
- [Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA](https://mlsanity.com/paper/2603.14782/) (2026)
- [MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries](https://mlsanity.com/paper/2505.16631/) (2025)
- [textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language](https://mlsanity.com/paper/2603.28213/) (2026)
- [Resource-Lean Lexicon Induction for German Dialects](https://mlsanity.com/paper/2604.23824/) (2026)

## Abstract (arXiv)

A large amount of local and culture-specific knowledge (e.g., people, traditions, food) can only be found in documents written in dialects. While there has been extensive research conducted on cross-lingual information retrieval (CLIR), the field of cross-dialect retrieval (CDIR) has received limited attention. Dialect retrieval poses unique challenges due to the limited availability of resources to train retrieval models and the high variability in non-standardized languages. We study these challenges on the example of German dialects and introduce the first German dialect retrieval dataset, dubbed WikiDIR, which consists of seven German dialects extracted from Wikipedia. Using WikiDIR, we demonstrate the weakness of lexical methods in dealing with high lexical variation in dialects. We further show that commonly used zero-shot cross-lingual transfer approach with multilingual encoders do not transfer well to extremely low-resource setups, motivating the need for resource-lean and dialect-specific retrieval models. We finally demonstrate that (document) translation is an effective way to reduce the dialect gap in CDIR.