mlsanityData to 7 Oct 2026

Measuring and Addressing Indexical Bias in Information Retrieval

arXiv 2406.04298 ↗First posted 6 Jun 2024cs.IR cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas. Consequently, IR can often introduce indexical biases, or biases in the positional order of documents. Although indexical bias can demonstrably affect people's opinion, voting patterns, and other behaviors, these issues remain understudied as the field lacks reliable metrics and procedures for automatically measuring indexical bias. Towards this end, we introduce the PAIR framework, which supports automatic bias audits for ranked documents or entire IR systems. After introducing DUO, the first general-purpose automatic bias metric, we run an extensive evaluation of 8 IR systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k queries spanning 1.4k controversial issue topics. A human behavioral study validates our approach, showing that our bias metric can help predict when and how indexical bias will shift a reader's opinion.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 31 of its 77 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

31 of 77 references.

References · 31

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  7. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  8. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  9. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  10. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  11. 2018Fairness of Exposure in Rankingscited by 189
  12. 2018Equity of Attention: Amortizing Individual Fairness in Rankingscited by 124
19 more
  1. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  2. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  3. 2016Measuring Fairness in Ranked Outputscited by 81
  4. 2021Fairness in Rankings and Recommendations: An Overviewcited by 62
  5. 2020Evaluating Stochastic Rankings with Expected Exposurecited by 53
  6. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  7. 2020Detecting Stance in Media on Global Warmingcited by 48
  8. 2019A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spacescited by 36
  9. 2021Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankerscited by 34
  10. 2020Do Neural Ranking Models Intensify Gender Bias?cited by 28
  11. 2019Quantifying the Impact of User Attention on Fair Group Representation in Ranked Listscited by 24
  12. 2021When Fair Ranking Meets Uncertain Inferencecited by 22
  13. 2017Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Mediacited by 18
  14. 2020Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentationcited by 18
  15. 2022Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Resultscited by 17
  16. 2022Evaluation Metrics for Measuring Bias in Search Engine Resultscited by 12
  17. 2024Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awarenesscited by 12
  18. 2016Modeling confirmation bias and polarizationcited by 11
  19. 2022A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevancecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 8
  2. 2025Linguistic Blind Spots of Large Language Modelscited by 6
  3. 2025EqualizeIR: Mitigating Linguistic Biases in Retrieval Modelscited by 5
  4. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  5. 2025Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrievalcited by 1
  6. 2026Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learningcited by 0