mlsanityData to 14 Oct 2026

Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos

arXiv 2601.06931 ↗First posted 11 Jan 2026cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a face-only counterfactual evaluation paradigm that isolates demographic effects while preserving real-image realism. Starting from real photographs, we generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed. Based on this paradigm, we construct FOCUS, a dataset of 480 scene-matched counterfactual images across six occupations and ten demographic groups, and propose REFLECT, a benchmark comprising three decision-oriented tasks: two-alternative forced choice, multiple-choice socioeconomic inference, and numeric salary recommendation. Experiments on five state-of-the-art VLMs reveal that demographic disparities persist under strict visual control and vary substantially across task formulations. These findings underscore the necessity of controlled, counterfactual audits and highlight task design as a critical factor in evaluating social bias in multimodal models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 44 references that are in the map. First posted on arXiv 11 Jan 2026.

Builds on

29 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Social Debiasing for Fair Multi-modal LLMs13 Aug
  2. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals30 May
  3. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb
  4. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples30 Nov
  5. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models3 Jul

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  4. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  5. 2017Counterfactual Fairnesscited by 804
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  8. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  9. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  11. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
12 more
  1. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  2. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  3. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  4. 2023The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendationscited by 41
  5. 2024"You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendationscited by 33
  6. 2023Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scalecited by 24
  7. 2022On Modality Bias Recognition and Reductioncited by 20
  8. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 18
  9. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  10. 2024Diversity-Aware k-Maximum Inner Product Search Revisitedcited by 7
  11. 2025PRISM: A Framework for Producing Interpretable Political Bias Embeddings with Political-Aware Cross-Encodercited by 5
  12. 2025Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrievalcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMscited by 1
  2. 2026GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Probing Intersectional Biases in Vision-Language Models with Counterfactual Examples
  2. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples
  3. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  4. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models
  5. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  6. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
  7. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  8. 2025Visual Cues of Gender and Race are Associated with Stereotyping in Vision-Language Models