mlsanityData to 7 Oct 2026

A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

arXiv 2410.07593 ↗First posted 10 Oct 2024cs.CV cs.AIcited by 18

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 46 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

32 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication24 Apr
  2. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  3. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  4. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
16 more
  1. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  2. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  3. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  4. 2018An Intersectional Definition of Fairnesscited by 126
  5. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  6. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  7. 2019FairFace: Face Attribute Dataset for Balanced Race, Gender, and Agecited by 87
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  9. 2018Hyperparameters and Tuning Strategies for Random Forestcited by 83
  10. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 76
  11. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  12. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  13. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30
  14. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSscited by 25
  15. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  16. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  2. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  3. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  4. 2025PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projectioncited by 5
  5. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Modelscited by 4
  6. 2024FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classificationcited by 3
  7. 2025Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencodercited by 3
  8. 2025Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQAcited by 2
  9. 2025FAME: Fairness-aware Attention-modulated Video Editingcited by 2
  10. 2025How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositionscited by 1
  11. 2025Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Modelscited by 1
  12. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
6 more
  1. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  2. 2025Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervisioncited by 0
  3. 2026Rank-Constrained Adaptation for Reliable Real-World Performancecited by 0
  4. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0
  5. 2026GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Modelscited by 0
  6. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  2. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
  3. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddings
  4. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
  5. 2024Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations
  6. 2024Debiasing Vison-Language Models with Text-Only Training
  7. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  8. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models