mlsanityData to 7 Oct 2026

Refusal as Silence: Gendered Disparities in Vision-Language Model Responses

arXiv 2406.08222 ↗First posted 12 Jun 2024cs.CV cs.AI cs.CYcited by 1

Not on the map. Why?

Abstract (arXiv)

Refusal behavior by Large Language Models is increasingly visible in content moderation, yet little is known about how refusals vary by the identity of the user making the request. This study investigates refusal as a sociotechnical outcome through a counterfactual persona design that varies gender identity--including male, female, non-binary, and transgender personas--while keeping the classification task and visual input constant. Focusing on a vision-language model (GPT-4V), we examine how identity-based language cues influence refusal in binary gender classification tasks. We find that transgender and non-binary personas experience significantly higher refusal rates, even in non-harmful contexts. Our findings also provide methodological implications for equity audits and content analysis using LLMs. Our findings underscore the importance of modeling identity-driven disparities and caution against uncritical use of AI systems for content coding. This study advances algorithmic fairness by reframing refusal as a communicative act that may unevenly regulate epistemic access and participation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 59 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

11 of 59 references.

References · 11

  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2016The Mythos of Model Interpretabilitycited by 716
  5. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  6. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  7. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  8. 2021Revealing Persona Biases in Dialogue Systemscited by 33
  9. 2019Assessing the Local Interpretability of Machine Learning Modelscited by 23
  10. 2023Smiling Women Pitching Down: Auditing Representational and Presentational Gender Biases in Image Generative AIcited by 19
  11. 2023All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026How Humans and LLMs Read Gender into "Gender-Neutral" Physical Descriptionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
  2. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models
  3. 2024Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs
  4. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
  5. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles
  6. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models
  7. 2025Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution
  8. 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models