mlsanityData to 7 Oct 2026

PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection

arXiv 2507.08979 ↗First posted 11 Jul 2025cs.CV cs.LGcited by 5

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

We introduce Projection-based Reduction of Implicit Spurious bias in vision-language Models (PRISM), a new data-free and task-agnostic solution for bias mitigation in VLMs like CLIP. VLMs often inherit and amplify biases in their training data, leading to skewed predictions. PRISM is designed to debias VLMs without relying on predefined bias categories or additional external data. It operates in two stages: first, an LLM is prompted with simple class prompts to generate scene descriptions that contain spurious correlations. Next, PRISM uses our novel contrastive-style debiasing loss to learn a projection that maps the embeddings onto a latent space that minimizes spurious correlations while preserving the alignment between image and text embeddings.Extensive experiments demonstrate that PRISM outperforms current debiasing methods on the commonly used Waterbirds and CelebA datasets We make our code public at: https://github.com/MahdiyarMM/PRISM.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 33 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

26 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddings7 Nov
  2. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks10 Oct
  3. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  9. 2019Learning De-biased Representations with Biased Representationscited by 199
  10. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  11. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  12. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
11 more
  1. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  2. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  3. 2022Gender Artifacts in Visual Datasetscited by 38
  4. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  5. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSscited by 25
  6. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  7. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  8. 2024Understanding Bias in Large-Scale Visual Datasetscited by 15
  9. 2023Investigating the Limitation of CLIP Models: The Worst-Performing Categoriescited by 12
  10. 2021Towards Learning an Unbiased Classifier from Biased Data via Conditional Adversarial Debiasingcited by 6
  11. 2024BEADs: Bias Evaluation Across Domainscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Modelscited by 1
  2. 2025Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Modelscited by 0
  3. 2026Zero-Shot Distracted Driver Detection via Vision Language Models with Double Decouplingcited by 0
  4. 2026Density-Aware Translation of Spurious Correlations in Zero-Shot VLMscited by 0
  5. 2026GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  2. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  3. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models
  4. 2026PRiSM: Prototype Regularization for Few-Shot VLMs
  5. 2026BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  8. 2024Debiasing Vison-Language Models with Text-Only Training