# Language Fairness in Multilingual Information Retrieval

arXiv: https://arxiv.org/abs/2405.00978 · HTML: https://mlsanity.com/paper/2405.00978/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 13 of 13)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [FA*IR: A Fair Top-k Ranking Algorithm](https://mlsanity.com/paper/1706.06368/) (2017)
- [Controlling Fairness and Bias in Dynamic Learning-to-Rank](https://mlsanity.com/paper/2005.14713/) (2020)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [Fairness in Ranking: A Survey](https://mlsanity.com/paper/2103.14000/) (2021)
- [Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists](https://mlsanity.com/paper/1901.10437/) (2019)
- [Neural Approaches to Multilingual Information Retrieval](https://mlsanity.com/paper/2209.01335/) (2022)
- [Soft Prompt Decoding for Multilingual Dense Retrieval](https://mlsanity.com/paper/2305.09025/) (2023)

## Built on by (most cited first, 5 of 5)

- [Investigating Language Preference of Multilingual RAG Systems](https://mlsanity.com/paper/2502.11175/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods](https://mlsanity.com/paper/2509.06195/) (2025)
- [All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG](https://mlsanity.com/paper/2604.20199/) (2026)
- [Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities](https://mlsanity.com/paper/2610.00606/) (2026)

## Abstract (arXiv)

Multilingual information retrieval (MLIR) considers the problem of ranking documents in several languages for a query expressed in a language that may differ from any of those languages. Recent work has observed that approaches such as combining ranked lists representing a single document language each or using multilingual pretrained language models demonstrate a preference for one language over others. This results in systematic unfair treatment of documents in different languages. This work proposes a language fairness metric to evaluate whether documents across different languages are fairly ranked through statistical equivalence testing using the Kruskal-Wallis test. In contrast to most prior work in group fairness, we do not consider any language to be an unprotected group. Thus our proposed measure, PEER (Probability of EqualExpected Rank), is the first fairness metric specifically designed to capture the language fairness of MLIR systems. We demonstrate the behavior of PEER on artificial ranked lists. We also evaluate real MLIR systems on two publicly available benchmarks and show that the PEER scores align with prior analytical findings on MLIR fairness. Our implementation is compatible with ir-measures and is available at http://github.com/hltcoe/peer_measure.