mlsanityData to 14 Oct 2026

Retrieval Sensitivity to Identity Signals in Queries

arXiv 2609.36534 ↗First posted 29 Sep 2026cs.CL cs.CY cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Dense retrievers decide which documents reach users and the language models that use them, yet they are typically evaluated with neutral queries. We ask whether the identity signals that real users express in their queries---political ideology and dialect---bias what a retriever returns. We design evaluations in two domains, political news and consumer-health questions, each pairing a controlled synthetic set that varies only the identity signal with naturalistic queries. Across five dense retrievers and a sparse baseline, every retriever (i) retrieves articles that align with the query's own political lean and (ii) performs worse for questions written in African American Language (AAL) than in White Mainstream English (WME). Two analyses tie these gaps to queries' identity signals beyond surface vocabulary: partialling out an aggregate lexical-asymmetry score leaves the synthetic gaps largely intact, and linear probes recover lean and dialect from the retrievers' query embeddings beyond token-level features. Left unaddressed, such retrieval biases risk contributing to polarization and reinforcing the health disparities already faced by AAL speakers. Code is available at https://github.com/Andrewtcr/bias-ret.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

37 of 62 references.

References · 37

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  6. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  7. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  8. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  9. 2018Fairness of Exposure in Rankingscited by 190
  10. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  11. 2020Embedding-based Retrieval in Facebook Searchcited by 156
  12. 2017FA*IR: A Fair Top-k Ranking Algorithmcited by 113
25 more
  1. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  2. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 88
  3. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  4. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  5. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 51
  6. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  7. 2021Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankerscited by 35
  8. 2021Evaluating the Robustness of Retrieval Pipelines with Query Variation Generatorscited by 34
  9. 2024Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systemscited by 26
  10. 2020MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Modelscited by 23
  11. 2022Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellingscited by 23
  12. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  13. 2022Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Resultscited by 17
  14. 2019ReQA: An Evaluation for End-to-End Answer Retrieval Modelscited by 16
  15. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  16. 2025LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are Highcited by 12
  17. 2024Examining the Influence of Political Bias on Large Language Model Performance in Stance Classificationcited by 9
  18. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generationcited by 9
  19. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  20. 2023Qbias -- A Dataset on Media Bias in Search Queries and Query Suggestionscited by 6
  21. 2025An Empirical Study of Position Bias in Modern Information Retrievalcited by 6
  22. 2024High Risk of Political Bias in Black Box Emotion Inference Modelscited by 2
  23. 2025A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systemscited by 2
  24. 2025Synergizing Implicit and Explicit User Interests: A Multi-Embedding Retrieval Framework at Pinterestcited by 1
  25. 2026Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurtscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles
  2. 2026Exploratory As-Analyzed No-Detection of Culturally-Marked Predicate-Triggered PII Amplification in a Synthetic-English RAG Probe: A Predicate-Resource-Confounded Audit
  3. 2026Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models
  4. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Bias
  5. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals
  6. 2025Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
  7. 2025Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes
  8. 2024The Political Preferences of LLMs