# Measuring and Addressing Indexical Bias in Information Retrieval

arXiv: https://arxiv.org/abs/2406.04298 · HTML: https://mlsanity.com/paper/2406.04298/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 77 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://mlsanity.com/paper/2010.05338/) (2020)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [Fairness in Rankings and Recommendations: An Overview](https://mlsanity.com/paper/2104.05994/) (2021)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval](https://mlsanity.com/paper/2009.13013/) (2020)
- [Detecting Stance in Media on Global Warming](https://mlsanity.com/paper/2010.15149/) (2020)
- [A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces](https://mlsanity.com/paper/1909.06092/) (2019)
- [Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankers](https://mlsanity.com/paper/2104.13640/) (2021)
- [Do Neural Ranking Models Intensify Gender Bias?](https://mlsanity.com/paper/2005.00372/) (2020)
- [Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists](https://mlsanity.com/paper/1901.10437/) (2019)
- [When Fair Ranking Meets Uncertain Inference](https://mlsanity.com/paper/2105.02091/) (2021)
- [Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media](https://mlsanity.com/paper/1704.01347/) (2017)
- [Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation](https://mlsanity.com/paper/2009.13815/) (2020)
- [Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Results](https://mlsanity.com/paper/2201.07754/) (2022)
- [Evaluation Metrics for Measuring Bias in Search Engine Results](https://mlsanity.com/paper/2210.10517/) (2022)
- [Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness](https://mlsanity.com/paper/2405.02714/) (2024)
- [Modeling confirmation bias and polarization](https://mlsanity.com/paper/1607.00022/) (2016)

## Built on by (most cited first, 6 of 6)

- [BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models](https://mlsanity.com/paper/2510.00307/) (2025)
- [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025)
- [EqualizeIR: Mitigating Linguistic Biases in Retrieval Models](https://mlsanity.com/paper/2504.07115/) (2025)
- [Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval](https://mlsanity.com/paper/2512.00804/) (2025)
- [Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization](https://mlsanity.com/paper/2503.07660/) (2025)
- [Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learning](https://mlsanity.com/paper/2603.12290/) (2026)

## Abstract (arXiv)

Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas. Consequently, IR can often introduce indexical biases, or biases in the positional order of documents. Although indexical bias can demonstrably affect people's opinion, voting patterns, and other behaviors, these issues remain understudied as the field lacks reliable metrics and procedures for automatically measuring indexical bias. Towards this end, we introduce the PAIR framework, which supports automatic bias audits for ranked documents or entire IR systems. After introducing DUO, the first general-purpose automatic bias metric, we run an extensive evaluation of 8 IR systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k queries spanning 1.4k controversial issue topics. A human behavioral study validates our approach, showing that our bias metric can help predict when and how indexical bias will shift a reader's opinion.