# Integrated Multimodal AI System for Retrieval-Augmented Reasoning, Object Sensing, and Damage Analysis

arXiv: https://arxiv.org/abs/2608.08935 · HTML: https://mlsanity.com/paper/2608.08935/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 5 of 5)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Knowledge Graph-Guided Retrieval Augmented Generation](https://mlsanity.com/paper/2502.06864/) (2025)
- [NIDA-CLIFGAN: Natural Infrastructure Damage Assessment through Efficient Classification Combining Contrastive Learning, Information Fusion and Generative Adversarial Networks](https://mlsanity.com/paper/2110.14518/) (2021)
- [Effective Damage Data Generation by Fusing Imagery with Human Knowledge Using Vision-Language Models](https://mlsanity.com/paper/2508.01380/) (2025)
- [Investigating Vision-Language Model for Point Cloud-based Vehicle Classification](https://mlsanity.com/paper/2504.08154/) (2025)

## Abstract (arXiv)

This work presents a unified multimodal AI system for damage assessment that integrates retrieval-augmented generation (RAG) models, thermal spectrum perception, vision foundation model pipelines, and exploratory wireless signal sensing. A RAG component is developed to ground a locally hosted language model in project-specific documentation, including specialized damage level classification criteria to mitigate hallucinations during inference. Controlled comparisons against static few-shot prompting demonstrate that dynamic retrieval improves grounding and factual consistency. We further compare vector-based RAG with a knowledge graph variant constructed via entity-relation extraction, and show that graph-based retrieval produces stronger responses for damage assessment queries requiring cross-document reasoning, motivating hybrid dense, sparse, and graph-aware retrieval. To address limitations of EO imagery under adverse lighting and weather conditions, infrared (IR)/thermal sensing is employed for object detection and segmentation. Our detectors generate candidate detections, yielding improved segmentation of a broad array of objects. Paired IR versus visible spectrum tracking experiments reveal failure modes, motivating multimodal fusion for robust object detection and damage analysis. Vision foundation and vision-language models are leveraged to generate synthetic damage imagery and classify damage severity with high accuracy, supporting training and validation of downstream damage assessment models. Finally, exploratory Wireless-based sensing demonstrates potential to detect presence, motion, and post-event environmental changes where EO and IR sensing are ineffective.