mlsanityData to 14 Oct 2026

SafeCoT: Improving VLM Safety with Minimal Reasoning

arXiv 2506.08399 ↗First posted 10 Jun 2025cs.AI cs.LGcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Ensuring safe and appropriate responses from vision-language models (VLMs) remains a critical challenge, particularly in high-risk or ambiguous scenarios. We introduce SafeCoT, a lightweight, interpretable framework that leverages rule-based chain-of-thought (CoT) supervision to improve refusal behavior in VLMs. Unlike prior methods that rely on large-scale safety annotations or complex modeling, SafeCoT uses minimal supervision to help models reason about safety risks and make context-aware refusals. Experiments across multiple benchmarks show that SafeCoT significantly reduces overrefusal and enhances generalization, even with limited training data. Our approach offers a scalable solution for aligning VLMs with safety-critical objectives.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 28 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

21 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  11. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  12. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
8 more
  1. 2024Multimodal Situational Safetycited by 58
  2. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  3. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
  4. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
  5. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  6. 2024Safety Alignment for Vision Language Modelscited by 28
  7. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Promptingcited by 8
  8. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Modelscited by 6
  2. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  3. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization
  2. 2026Visual Grounding Safety in Vision-Language Models
  3. 2026Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
  4. 2026EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
  5. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMs
  6. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  7. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety
  8. 2026Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models