mlsanityData to 14 Oct 2026

Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

arXiv 2609.05540 ↗First posted 2 Sep 2026cs.CV cs.AIcited by 0

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 11 of its 57 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

11 of 57 references.

References · 11

  1. 2025Vision Language Models are Biasedcited by 66
  2. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  3. 2025Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromisecited by 8
  4. 2024Debiasing Large Vision-Language Models by Ablating Protected Attribute Representationscited by 6
  5. 2025EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognitioncited by 6
  6. 2025Vision-Language Models display a strong gender biascited by 5
  7. 2025LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editingcited by 5
  8. 2022AU-Aware Vision Transformers for Biased Facial Expression Recognitioncited by 4
  9. 2025Causal-Ex: Causal Graph-based Micro and Macro Expression Spottingcited by 3
  10. 2025Compound Expression Recognition via Large Vision-Language Modelscited by 3
  11. 2021Weakly Supervised Learning for Facial Affective Behavior Analysis : A Reviewcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
  2. 2026Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
  3. 2026MIRAGE: The Illusion of Visual Understanding
  4. 2026When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models
  5. 2026EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
  6. 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
  7. 2026FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
  8. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models