mlsanityData to 7 Oct 2026

BendVLM: Test-Time Debiasing of Vision-Language Embeddings

arXiv 2411.04420 ↗First posted 7 Nov 2024cs.CV cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-language model (VLM) embeddings have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. VLMs are being quickly adopted for a variety of tasks ranging from few-shot classification to text-guided image generation, making debiasing VLM embeddings crucial. Debiasing approaches that fine-tune the VLM often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a "one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose Bend-VLM, a nonlinear, fine-tuning-free approach for VLM embedding debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference time, making our method more appropriate for online, open-set tasks such as retrieval and text guided image generation.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

50 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Identifying Implicit Social Biases in Vision-Language Models1 Nov
  2. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  3. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  4. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2023Mistral 7Bcited by 3,065
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
34 more
  1. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  3. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  4. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  5. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  6. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 192
  7. 2020Towards Debiasing Sentence Representationscited by 170
  8. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 149
  9. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  10. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  11. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  12. 2023Finetuning Text-to-Image Diffusion Models for Fairnesscited by 90
  13. 2022The Fallacy of AI Functionalitycited by 88
  14. 2019FairFace: Face Attribute Dataset for Balanced Race, Gender, and Agecited by 87
  15. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  16. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  17. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  18. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  19. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  20. 2020Mitigating Gender Bias in Captioning Systemscited by 50
  21. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  22. 2023Prompting Visual-Language Models for Dynamic Facial Expression Recognitioncited by 43
  23. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  24. 2022Measuring Representational Harms in Image Captioningcited by 36
  25. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  26. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30
  27. 2022Markedness in Visual Semantic AIcited by 29
  28. 2022ADEPT: A DEbiasing PrompT Frameworkcited by 20
  29. 2023Debiased Fine-Tuning for Vision-language Models by Prompt Regularizationcited by 20
  30. 2023Mitigating Test-Time Bias for Fair Image Retrievalcited by 20
  31. 2022Robots Enact Malignant Stereotypescited by 14
  32. 2023Evaluating the Fairness of Discriminative Foundation Models in Computer Visioncited by 12
  33. 2024Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Modecited by 6
  34. 2023An Investigation of Representation and Allocation Harms in Contrastive Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 8.

17 papers.

Sort:

By area:

  1. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
  2. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  3. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  4. 2025Enhancing Concept Localization in CLIP-based Concept Bottleneck Modelscited by 6
  5. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  6. 2025PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projectioncited by 5
  7. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  8. 2025Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alonecited by 1
  9. 2025SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentationcited by 1
  10. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
  11. 2026SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Modelscited by 1
  12. 2025SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networkscited by 0
5 more
  1. 2026Rank-Constrained Adaptation for Reliable Real-World Performancecited by 0
  2. 2026Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositionscited by 0
  3. 2026Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decodingcited by 0
  4. 2026Fairness-Aware Test-Time Prompt Tuningcited by 0
  5. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
  2. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
  3. 2024Debiasing Vison-Language Models with Text-Only Training
  4. 2024Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations
  5. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  6. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  7. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
  8. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models