mlsanityData to 7 Oct 2026

SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detection

arXiv 2512.11215 ↗First posted 12 Dec 2025cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Wildfire smoke is transparent, amorphous, and often visually confounded with clouds, making early-stage detection particularly challenging. In this work, we introduce a benchmark, called SmokeBench, to evaluate the ability of multimodal large language models (MLLMs) to recognize and localize wildfire smoke in images. The benchmark consists of four tasks: (1) smoke classification, (2) tile-based smoke localization, (3) grid-based smoke localization, and (4) smoke detection. We evaluate several MLLMs, including Idefics2, Qwen2.5-VL, InternVL3, Unified-IO 2, Grounding DINO, GPT-4o, and Gemini-2.5 Pro. Our results show that while some models can classify the presence of smoke when it covers a large area, all models struggle with accurate localization, especially in the early stages. Further analysis reveals that smoke volume is strongly correlated with model performance, whereas contrast plays a comparatively minor role. These findings highlight critical limitations of current MLLMs for safety-critical wildfire monitoring and underscore the need for methods that improve early-stage smoke localization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 58 references that are in the map. First posted on arXiv 12 Dec 2025.

Builds on

32 of 58 references.

References · 32

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2017Mask R-CNNcited by 7,259
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
20 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  5. 2024What matters when building vision-language models?cited by 321
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  7. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  9. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  10. 2022Towards Open-Set Object Detection and Discoverycited by 44
  11. 2018Deep Object Co-Segmentationcited by 35
  12. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  13. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Modelscited by 20
  14. 2021FIgLib & SmokeyNet: Dataset and Deep Learning Model for Real-Time Wildland Fire Smoke Detectioncited by 11
  15. 2023Transmission-Guided Bayesian Generative Model for Smoke Segmentationcited by 6
  16. 2020Project RISE: Recognizing Industrial Smoke Emissionscited by 5
  17. 2024Revisiting Multi-Modal LLM Evaluationcited by 5
  18. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  19. 2024A transformer boosted UNet for smoke segmentation in complex backgrounds in multispectral LandSat imagerycited by 3
  20. 2023A Survey on IoT Ground Sensing Systems for Early Wildfire Detection: Technologies, Challenges and Opportunitiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessitycited by 1
  2. 2026AusSmoke meets MultiNatSmoke: a fully-labelled diverse smoke segmentation datasetcited by 0
  3. 2026Distilling Vision-Language Models for On-Device Fire Understandingcited by 0
  4. 2026SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WildfireVLM: AI-powered Analysis for Early Wildfire Detection and Risk Assessment Using Satellite Imagery
  2. 2025DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding
  3. 2025MFGDiffusion: Mask-Guided Smoke Synthesis for Enhanced Forest Fire Detection
  4. 2026WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring
  5. 2026FlameVQA: A Physically-Grounded UAV Wildfire VQA Benchmark with Radiometric Thermal Supervision
  6. 2025Automated Wildfire Damage Assessment from Multi view Ground level Imagery Via Vision Language Models
  7. 2025WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada
  8. 20263D Smoke Scene Reconstruction Guided by Vision Priors from Multimodal Large Language Models