mlsanityData to 14 Oct 2026

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

arXiv 2609.07823 ↗First posted 7 Sep 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 36 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

12 of 36 references.

References · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  5. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  6. 2015BoWFire: Detection of Fire in Still Images by Integrating Pixel Color and Texture Analysiscited by 16
  7. 2025SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detectioncited by 5
  8. 2025SmokeBench: A Real-World Dataset for Surveillance Image Desmoking in Early-Stage Fire Scenescited by 4
  9. 2025FireSentry: A Multi-Modal Spatio-temporal Benchmark Dataset for Fine-Grained Wildfire Spread Forecastingcited by 3
  10. 2025DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understandingcited by 2
  11. 2024Benchmarking Multi-Scene Fire and Smoke Detectioncited by 1
  12. 2024Fire and Smoke Detection with Burning Intensity Representationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Distilling Vision-Language Models for On-Device Fire Understanding
  2. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  3. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  4. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
  5. 2025MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models
  6. 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
  7. 2025Robotic Fire Risk Detection based on Dynamic Knowledge Graph Reasoning: An LLM-Driven Approach with Graph Chain-of-Thought
  8. 2024FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models