mlsanityData to 14 Oct 2026

Mitigating Test-Time Bias for Fair Image Retrieval

arXiv 2305.19329 ↗First posted 23 May 2023cs.CV cs.IR cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at urlhttps://anonymous.4open.science/r/FairTextbasedImageRetrieval-D8B2.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

37 of 64 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  10. 2017Counterfactual Fairnesscited by 804
  11. 2014Certifying and removing disparate impactcited by 715
  12. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
25 more
  1. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  2. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  3. 2015Censoring Representations with an Adversarycited by 192
  4. 2023Debiasing Vision-Language Models via Biased Promptscited by 149
  5. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  6. 2020To be Robust or to be Fair: Towards Fairness in Adversarial Trainingcited by 137
  7. 2019Fairness in Recommendation Ranking through Pairwise Comparisonscited by 127
  8. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  9. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  10. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  11. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  12. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  13. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  14. 2020Controlling Fairness and Bias in Dynamic Learning-to-Rankcited by 68
  15. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  16. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  17. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  18. 2021Fairness for Image Generation with Uncertain Sensitive Attributescited by 31
  19. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparitiescited by 31
  20. 2020Mitigating Bias in Set Selection with Noisy Protected Attributescited by 29
  21. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  22. 2019Implicit Diversity in Image Summarizationcited by 18
  23. 2022Fair Ranking with Noisy Protected Attributescited by 17
  24. 2023Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Samplingcited by 12
  25. 2015Fairness Constraints: Mechanisms for Fair Classificationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 6; 2026: 3.

20 papers.

Sort:

By area:

  1. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  2. 2024On the Challenges and Opportunities in Generative AIcited by 31
  3. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  4. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 18
  5. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  6. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 13
  7. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  8. 2024Organizing Unstructured Image Collections using Natural Languagecited by 7
  9. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  10. 2025Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomescited by 6
  11. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  12. 2023ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlationscited by 5
8 more
  1. 2025Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4cited by 5
  2. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  3. 2025Mutual Learning for Hashing: Unlocking Strong Hash Functions from Weak Supervisioncited by 3
  4. 2024Multi-Group Proportional Representation in Retrievalcited by 2
  5. 2023UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimationcited by 1
  6. 2024A Note on Bias to Completecited by 1
  7. 2026UniHash: Unifying Pointwise and Pairwise Hashing Paradigmscited by 1
  8. 2026ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models
  2. 2024FairRAG: Fair Human Generation via Fair Retrieval Augmentation
  3. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs
  4. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
  5. 2025BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
  6. 2025Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search
  7. 2025Mitigating Bias in RAG: Controlling the Embedder
  8. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts