mlsanityData to 7 Oct 2026

FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization

arXiv 2210.14562 ↗First posted 26 Oct 2022cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided into two steps: Attribute Prototype Learning (APL) and Representation Neutralization (RN). In the first step, we extract the concepts needed for debiasing in CLIP. We use the query with learnable word vector prefixes as the extraction structure. In the second step, we first divide the attributes into target and bias attributes. By analysis, we find that both attributes have an impact on the bias. Therefore, we try to eliminate the bias by using Re-Representation Matrix (RRM) to achieve the neutralization of the representation. We compare the debiasing effect and retrieval performance with other methods, and experiments demonstrate that FairCLIP can achieve the best compatibility. Although FairCLIP is used to eliminate bias in image retrieval, it achieves the neutralization of the representation which is common to all CLIP downstream tasks. This means that FairCLIP can be applied as a general debiasing method for other fairness issues related to CLIP.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

27 of 32 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  12. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
15 more
  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  3. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 137
  4. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  5. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  6. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  7. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  8. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  9. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  10. 2022CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIPcited by 60
  11. 2020DeVLBert: Learning Deconfounded Visio-Linguistic Representationscited by 42
  12. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  13. 2020Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasingcited by 32
  14. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  15. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 8; 2025: 3; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  2. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  3. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  4. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Setscited by 27
  5. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSscited by 25
  6. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  7. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  8. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  9. 2023Mitigating Test-Time Bias for Fair Image Retrievalcited by 20
  10. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Taskscited by 18
  11. 2024Harm Amplification in Text-to-Image Modelscited by 14
  12. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 10
8 more
  1. 2024Nearest Neighbor Normalization Improves Multimodal Retrievalcited by 7
  2. 2023Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Modelscited by 6
  3. 2025PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projectioncited by 5
  4. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Modelscited by 4
  5. 2023Which One? Leveraging Context Between Objects and Multiple Views for Language Groundingcited by 3
  6. 2024Multi-Group Proportional Representation in Retrievalcited by 2
  7. 2024Cross-modality debiasing: using language to mitigate sub-population shifts in imagingcited by 1
  8. 2026Bias Redistribution in Visual Machine Unlearning: Does Forgetting One Group Harm Another?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP
  2. 2024Identifying Implicit Social Biases in Vision-Language Models
  3. 2025FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models
  4. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
  5. 2026Interpretable Debiasing of Vision-Language Models for Social Fairness
  6. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication
  7. 2025BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
  8. 2025Mitigating Bias in RAG: Controlling the Embedder