# VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

arXiv: https://arxiv.org/abs/2509.21609 · HTML: https://mlsanity.com/paper/2509.21609/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges](https://mlsanity.com/paper/2102.12219/) (2021)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)
- [xBD: A Dataset for Assessing Building Damage from Satellite Imagery](https://mlsanity.com/paper/1911.09296/) (2019)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response](https://mlsanity.com/paper/2505.21089/) (2025)
- [Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph](https://mlsanity.com/paper/2107.11970/) (2021)
- [InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation](https://mlsanity.com/paper/2305.06002/) (2023)
- [VQA-Aid: Visual Question Answering for Post-Disaster Damage Assessment and Analysis](https://mlsanity.com/paper/2106.10548/) (2021)
- [Improving Image Captioning by Leveraging Knowledge Graphs](https://mlsanity.com/paper/1901.08942/) (2019)
- [Comparative Analysis of Word Embeddings for Capturing Word Similarities](https://mlsanity.com/paper/2005.03812/) (2020)
- [Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction](https://mlsanity.com/paper/2402.17969/) (2024)
- [LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description](https://mlsanity.com/paper/2408.04957/) (2024)
- [DeepDamageNet: A two-step deep-learning model for multi-disaster building damage segmentation and classification using satellite imagery](https://mlsanity.com/paper/2405.04800/) (2024)
- [Using ConceptNet to Teach Common Sense to an Automated Theorem Prover](https://mlsanity.com/paper/1912.12957/) (2019)
- [Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning](https://mlsanity.com/paper/2310.14785/) (2023)

## Abstract (arXiv)

General-purpose vision-language models (VLMs) such as LLaVA and QwenVL produce descriptions of disaster imagery that lack domain-specific vocabulary and actionable detail. We propose the Vision-Language Caption Enhancer (VLCE), a framework that integrates external semantic knowledge from ConceptNet and WordNet into the caption generation process for post-disaster satellite and UAV imagery. VLCE operates in two stages: first, a baseline VLM generates an initial caption conditioned on YOLOv8 object detections; second, a knowledge-enriched sequential model, a CNN-LSTM or a hierarchical cross-modal Transformer, refines the caption using a vocabulary augmented with 1,566 domain-relevant terms extracted from knowledge graphs. We evaluate VLCE on two disaster benchmarks: xBD (satellite, 6,369 images, 3 damage classes) and RescueNet (UAV, 4,494 images, 12 damage classes), using CLIPScore for semantic alignment and InfoMetIC for informativeness. On RescueNet with the Transformer decoder, VLCE with knowledge graph enrichment produces captions preferred over QwenVL baselines in 95.33% of image pairs on InfoMetIC and 73.64% on CLIPScore. Qualitative analysis shows that without knowledge graph integration, generated captions exhibit hallucinations, word repetition, and semantic incoherence, whereas knowledge-enriched captions maintain factual consistency and domain-appropriate vocabulary.