# Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

arXiv: https://arxiv.org/abs/2609.05540 · HTML: https://mlsanity.com/paper/2609.05540/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 11 of 11)

- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)](https://mlsanity.com/paper/2410.19314/) (2024)
- [Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromise](https://mlsanity.com/paper/2510.17372/) (2025)
- [Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations](https://mlsanity.com/paper/2410.13976/) (2024)
- [EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition](https://mlsanity.com/paper/2505.20033/) (2025)
- [Vision-Language Models display a strong gender bias](https://mlsanity.com/paper/2508.11262/) (2025)
- [LaTo: Landmark-tokenized Diffusion Transformer for Fine-grained Human Face Editing](https://mlsanity.com/paper/2509.25731/) (2025)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Causal-Ex: Causal Graph-based Micro and Macro Expression Spotting](https://mlsanity.com/paper/2503.09098/) (2025)
- [Compound Expression Recognition via Large Vision-Language Models](https://mlsanity.com/paper/2503.11241/) (2025)
- [Weakly Supervised Learning for Facial Affective Behavior Analysis : A Review](https://mlsanity.com/paper/2101.09858/) (2021)

## Abstract (arXiv)

Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.