mlsanityData to 7 Oct 2026

SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

arXiv 2408.10202 ↗First posted 19 Aug 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Large-scale vision-language models, such as CLIP, are known to contain societal bias regarding protected attributes (e.g., gender, age). This paper aims to address the problems of societal bias in CLIP. Although previous studies have proposed to debias societal bias through adversarial learning or test-time projecting, our comprehensive study of these works identifies two critical limitations: 1) loss of attribute information when it is explicitly disclosed in the input and 2) use of the attribute annotations during debiasing process. To mitigate societal bias in CLIP and overcome these limitations simultaneously, we introduce a simple-yet-effective debiasing method called SANER (societal attribute neutralizer) that eliminates attribute information from CLIP text features only of attribute-neutral descriptions. Experimental results show that SANER, which does not require attribute annotations and preserves original information for attribute-specific descriptions, demonstrates superior debiasing ability than the existing methods.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 60 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

49 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment20 Jun
  2. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  3. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  4. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities26 Jan
  5. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
32 more
  1. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  2. 2021Image Segmentation Using Text and Image Promptscited by 509
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  7. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  8. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  9. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 149
  10. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 137
  11. 2020Fair Attribute Classification through Latent Space De-biasingcited by 125
  12. 2022How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?cited by 92
  13. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  14. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  15. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  16. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  17. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  18. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  19. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  20. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  21. 2022Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Biascited by 43
  22. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  23. 2021Measuring Fairness in Generative Modelscited by 31
  24. 2022Markedness in Visual Semantic AIcited by 29
  25. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  26. 2022VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioningcited by 25
  27. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  28. 2022VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioningcited by 19
  29. 2024SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generationcited by 19
  30. 2023Ethical Considerations for Responsible Data Curationcited by 17
  31. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  32. 2024Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
  2. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  3. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  4. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  5. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  6. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  7. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferencescited by 2
  8. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  9. 2025Prompting Forgetting: Unlearning in GANs via Textual Guidancecited by 1
  10. 2025Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Modelscited by 1
  11. 2026SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Modelscited by 1
  12. 2026Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimizationcited by 1
7 more
  1. 2025FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancingcited by 0
  2. 2026Rank-Constrained Adaptation for Reliable Real-World Performancecited by 0
  3. 2026Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategiescited by 0
  4. 2026Embedding Arithmetic: A Lightweight, Tuning-Free Framework for Post-hoc Bias Mitigation in Text-to-Image Modelscited by 0
  5. 2026Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositionscited by 0
  6. 2026Selective Test-Time Debiasing for CLIP via Reward Gatingcited by 0
  7. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations
  2. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization
  3. 2024Identifying Implicit Social Biases in Vision-Language Models
  4. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  5. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  6. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
  7. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs
  8. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks