mlsanityData to 14 Oct 2026

Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Bias

arXiv 2609.16651 ↗First posted 15 Sep 2026cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Behavioral benchmarking reveals what biases exist in vision-language models but not which internal components are most sensitive to targeted intervention, precluding principled intervention. We argue for mechanism-level evaluation as a necessary complement, demonstrating causal mediation analysis as a diagnostic instrument for gender bias. We decompose gender-cue effects into controlled indirect effects attributable to specific-layer activations and direct effects through all other pathways, producing layer-by-layer mechanistic signatures. Across six models spanning three architectural families (LLaVA-1.5, LLaVA-NeXT, InstructBLIP at 7B/13B) and two 8B-scale architectures, three findings emerge: language-layer activations exhibit the greatest output sensitivity under controlled intervention, with the direct component often carrying the opposite sign; architectural choices redistribute layer-wise sensitivity to activation replacement; and counterfactual scores diverge from surface-level scores, exposing implicit associations. Systematic ablation validates internal consistency. An intervention experiment finds that the average indirect effect (AIE) and downstream intervention effectiveness are only weakly correlated (Pearson r = 0.33), and the layer with the second-largest AIE produces near-zero bias change---indicating that mechanistic diagnosis captures activation-replacement sensitivity but does not, by itself, identify optimal intervention targets. These results show mechanism-level evaluation captures architecture-specific sensitivity patterns that behavioral benchmarks cannot; pairing both should become standard NLP practice. Code: https://github.com/zhaozhipeng1997/CARD-GenderBias.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 47 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

15 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Interpretable Debiasing of Vision-Language Models for Social Fairness27 Feb
  2. 2025BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models30 Nov
  3. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models26 May

Other references · 12

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  4. 2025Vision Language Models are Biasedcited by 66
  5. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 55
  6. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  7. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  8. 2026Locating Demographic Bias at the Attention-Head Level in CLIP's Vision Encodercited by 2
  9. 2026Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Studycited by 2
  10. 2026Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretabilitycited by 2
  11. 2025Pathological Truth Bias in Vision-Language Modelscited by 1
  12. 2026Mechanisms of Object Localization in Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
  2. 2026Vision-Language Models Suppress Female Representations Under Ambiguous Input
  3. 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Models
  4. 2026Gender Bias in Vision-Language In-Context Learning
  5. 2026ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models
  6. 2026Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts
  7. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities
  8. 2024More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models