Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQA
On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs
Abstract (arXiv)
Recent advances in large vision-language models (LVLMs) have amplified concerns about fairness, yet existing evaluations remain confined to demographic attributes and often conflate fairness with refusal behavior. This paper broadens the scope of fairness by introducing a counterfactual VQA benchmark that probes the decision boundaries of closed-source LVLMs under controlled context shifts. Each image pair differs in a single visual attribute that has been validated as irrelevant to the question, enabling ground-truth-free and refusal-aware analysis of reasoning stability. Comprehensive experiments reveal that non-demographic attributes, such as environmental context or social behavior, distort LVLM decision-making more strongly than demographic ones. Moreover, instruction-based debiasing shows limited effectiveness and can even amplify these asymmetries, whereas exposure to a small number of human norm validated examples from our benchmark encourages more consistent and balanced responses, highlighting its potential not only as an evaluative framework but also as a means for understanding and improving model behavior. Together, these results provide a practial basis for auditing contextual biases even in black-box LVLMs and contribute to more transparent and equitable multimodal reasoning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 5 Aug 2025.
Builds on
34 of 44 references.
Closest ancestors · 10
Same direction, newest first
- 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models10 Mar
- 2024Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations17 Oct
- 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks10 Oct
- 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing30 Jun
- 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts25 Jun
- 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model20 Jun
- 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models21 Feb
- 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples30 Nov
- 2023Survey of Social Bias in Vision-Language Models24 Sep
- 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
Other references · 24
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2025Qwen3 Technical Reportcited by 8,195
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2023A Survey on Multimodal Large Language Modelscited by 898
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2017Counterfactual Fairnesscited by 802
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
12 more
- 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
- 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
- 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
- 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
- 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
- 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
- 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
- 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
- 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
- 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
- 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
- 2023Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronounscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models
- 2026FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
- 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs
- 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
- 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation
- 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
- 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks
- 2025How Reasoning Influences Intersectional Biases in Vision Language Models