mlsanityData to 14 Oct 2026

VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

arXiv 2509.21609 ↗First posted 25 Sep 2025cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

General-purpose vision-language models (VLMs) such as LLaVA and QwenVL produce descriptions of disaster imagery that lack domain-specific vocabulary and actionable detail. We propose the Vision-Language Caption Enhancer (VLCE), a framework that integrates external semantic knowledge from ConceptNet and WordNet into the caption generation process for post-disaster satellite and UAV imagery. VLCE operates in two stages: first, a baseline VLM generates an initial caption conditioned on YOLOv8 object detections; second, a knowledge-enriched sequential model, a CNN-LSTM or a hierarchical cross-modal Transformer, refines the caption using a vocabulary augmented with 1,566 domain-relevant terms extracted from knowledge graphs. We evaluate VLCE on two disaster benchmarks: xBD (satellite, 6,369 images, 3 damage classes) and RescueNet (UAV, 4,494 images, 12 damage classes), using CLIPScore for semantic alignment and InfoMetIC for informativeness. On RescueNet with the Transformer decoder, VLCE with knowledge graph enrichment produces captions preferred over QwenVL baselines in 95.33% of image pairs on InfoMetIC and 73.64% on CLIPScore. Qualitative analysis shows that without knowledge graph integration, generated captions exhibit hallucinations, word repetition, and semantic incoherence, whereas knowledge-enriched captions maintain factual consistency and domain-appropriate vocabulary.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 39 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

19 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response27 May

Other references · 18

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  5. 2021Object Detection in Aerial Images: A Large-Scale Benchmark and Challengescited by 204
  6. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  7. 2019xBD: A Dataset for Assessing Building Damage from Satellite Imagerycited by 139
  8. 2019MMKG: Multi-Modal Knowledge Graphscited by 105
  9. 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
  10. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  11. 2021VQA-Aid: Visual Question Answering for Post-Disaster Damage Assessment and Analysiscited by 15
  12. 2019Improving Image Captioning by Leveraging Knowledge Graphscited by 11
6 more
  1. 2020Comparative Analysis of Word Embeddings for Capturing Word Similaritiescited by 9
  2. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7
  3. 2024LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Descriptioncited by 7
  4. 2024DeepDamageNet: A two-step deep-learning model for multi-disaster building damage segmentation and classification using satellite imagerycited by 5
  5. 2019Using ConceptNet to Teach Common Sense to an Automated Theorem Provercited by 3
  6. 2023Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Think First, Assign Next (ThiFAN-VQA): A Two-stage Chain-of-Thought Framework for Post-Disaster Damage Assessment
  2. 2026DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes
  3. 2026From Pixels to Semantics: A Multi-Stage AI Framework for Structural Damage Detection in Satellite Imagery
  4. 2026DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Response
  5. 2026Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation
  6. 2025Effective Damage Data Generation by Fusing Imagery with Human Knowledge Using Vision-Language Models
  7. 2025Explaining the Unseen: Multimodal Vision-Language Reasoning for Situational Awareness in Underground Mining Disasters
  8. 2026Satellite-to-Street: Synthesizing Post-Disaster Views from Satellite Imagery via Generative Vision Models