mlsanityData to 7 Oct 2026

RAPTOR-AI for Disaster OODA Loop: Hierarchical Multimodal RAG with Experience-Driven Agentic Decision-Making

arXiv 2602.00030 ↗First posted 18 Jan 2026cs.LGcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Humanitarian Assistance and Disaster Relief (HADR) operations demand rapid synthesis of multimodal information for time-critical decision-making under extreme uncertainty. Traditional information systems struggle with the fragmented, multimodal nature of disaster data and lack adaptive reasoning capabilities essential for dynamic emergency contexts. This work introduces RAPTOR-AI, an agentic multimodal Retrieval-Augmented Generation (RAG) framework that advances beyond conventional static knowledge bases by implementing dynamic, experience-driven decision support for disaster response. The system addresses HADR requirements across initial rescue, recovery, and reconstruction phases through three key innovations: hierarchical multimodal knowledge construction from diverse sources (textual reports, aerial imagery, historical documentation), entropy-aware agentic control that dynamically selects optimal retrieval strategies based on situational context, and experiential knowledge integration using LoRA adaptation for both expert and non-expert responders. The framework constructs hierarchical knowledge trees from 46 tsunami-related PDFs (2,378 pages) using BLIP-based image understanding, ColVBERT embeddings, and long-context summarization within the OODA loop (Observe, Orient, Decide, Act) tactical framework. Experiments demonstrate significant improvements over existing approaches: 23% improvement in retrieval precision, 31% better situational grounding, and 27% enhanced task decomposition accuracy, with efficient scaling up to 3,000 document chunks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 62 references that are in the map. First posted on arXiv 18 Jan 2026.

Builds on

35 of 62 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
23 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  13. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  14. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  15. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  16. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  17. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  18. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  19. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  20. 2018CrisisMMD: Multimodal Twitter Datasets from Natural Disasterscited by 88
  21. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  22. 2014Processing Social Media Messages in Mass Emergency: A Surveycited by 50
  23. 2017Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networkscited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation
  2. 2026Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
  3. 2026CrisiSense-RAG: Crisis Sensing Multimodal Retrieval-Augmented Generation for Rapid Disaster Impact Assessment
  4. 2025Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reports
  5. 2026DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments
  6. 2025Disaster Management in the Era of Agentic AI Systems: A Vision for Collective Human-Machine Intelligence for Augmented Resilience
  7. 2025Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports
  8. 2024Integration of Large Vision Language Models for Efficient Post-disaster Damage Assessment and Reporting