mlsanityData to 14 Oct 2026

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

arXiv 2603.19028 ↗First posted 19 Mar 2026cs.CV cs.AI cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

also in 1 more
Abstract (arXiv)

Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 33 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

26 of 33 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection11 Jul
  2. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddings7 Nov
  3. 2024Identifying Implicit Social Biases in Vision-Language Models1 Nov
  4. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias17 Oct
  5. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP19 Aug
  6. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  7. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  8. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  7. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  8. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  9. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  10. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  11. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  12. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
6 more
  1. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  2. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  3. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  4. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 18
  5. 2025Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifierscited by 4
  6. 2025Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alonecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Interpreting the linear structure of vision-language model embedding spaces
  2. 2026Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
  3. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  4. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models
  5. 2026BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models
  6. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models
  7. 2026Interpretable Debiasing of Vision-Language Models for Social Fairness
  8. 2025Steering CLIP's vision transformer with sparse autoencoders