mlsanityData to 7 Oct 2026

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

arXiv 2608.23313 ↗First posted 24 Aug 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view cannot tell whether a model is safe for the right multimodal reason. Safelooking behavior may reflect keyword-triggered refusal, missed visual hazards, or over-refusal of benign-sensitive inputs. We introduce EviSafe, an evidence-grounded framework for VLM safety that jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes in safety-critical evidence. EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios and 2,452 targeted counterfactual variants across eight safety domains and eight risk-source types. Each scenario includes a gold safety decision, evidence annotations, a safe-response policy, and counterfactual interventions. The three-probe protocol queries models with natural-response, evidencereporting, and counterfactual-response prompts, then scores them using an evidence-aware judge. Across eleven evaluated VLMs, natural severity accuracy ranges from 27.6% to 52.8%, relaxed diagnostic consistency from 6.1% to 29.3%, and unsafe-to-safe counterfactual transition success from 30.4% to 58.4%. These gaps show that the evaluated VLMs are not reliably safe for the right multimodal reason and motivate evaluation beyond refusal counts.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 38 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

29 of 38 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory16 Mar
  2. 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs3 Feb
  3. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization17 Nov
  4. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models12 Oct
  5. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  6. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  8. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 21

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
9 more
  1. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  2. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  3. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  4. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  5. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  6. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  7. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  8. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinkingcited by 3
  9. 2026How Sensitive Are Safety Benchmarks to Judge Configuration Choices?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A2Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Local Evidence to Safety Verdicts: Causal Tracing in Vision-Language Models
  2. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  3. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  4. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  5. 2025SIA: Enhancing Safety via Intent Awareness for Vision-Language Models
  6. 2026Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models
  7. 2025ELITE: Enhanced Language-Image Toxicity Evaluation for Safety
  8. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety