mlsanityData to 7 Oct 2026

Geospatial-Temporal Sensemaking of Remote Sensing Activity Detections with Multimodal Large Language Model

arXiv 2605.10739 ↗First posted 11 May 2026eess.IV cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We introduce SMART-HC-VQA, a Sentinel-2-based visual question answering dataset derived from the IARPA SMART Heavy Construction dataset, designed for spatiotemporal analysis of human activity. The dataset transforms construction-site annotations, construction-type labels, temporal-phase labels, geographic metadata, and observation relationships into natural language question-answer triplets. This approach redefines the existing dataset as a temporally extended automatic target recognition and visual question answering (VQA) challenge, considering a fixed geospatial site as a target whose attributes and activity states evolve across sparse satellite observations. Currently, SMART-HC-VQA comprises 21,837 accessible Sentinel-2 image chips, 65,511 single-image VQA examples, and approximately 2.3 million two-image temporal comparison examples generated via our novel Image-Pairwise Combinatorial Augmentation. We detail the workflow for retrieving and processing Sentinel-2 imagery, segmenting large satellite tiles into site-centered images, maintaining traceability to SMART-HC annotations, and analyzing the distributions of site size, observation count, temporal coverage, construction type, and phase labels. Additionally, we describe an implemented multi-image MLLM training framework based on LLaVA-NeXT Mistral-7B, adapted to accept multiple dated image inputs and train on metadata-derived VQA examples. This work offers a reproducible foundation for understanding language-guided remote sensing activities, aiming not only to detect change but also to reason about the ongoing processes, their progression, and potential future developments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 36 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

21 of 36 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026SAR-RAG: ATR Visual Question Answering by Semantic Search, Retrieval, and MLLM Generation4 Feb
  2. 2025BTCChat: Advancing Remote Sensing Bi-temporal Change Captioning with Multimodal Large Language Model7 Sep
  3. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution27 May
  4. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding27 May
  5. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understanding4 Apr
  6. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues19 Dec
  7. 2024IFShip: Interpretable Fine-grained Ship Classification with Domain Knowledge-Enhanced Vision-Language Models13 Aug
  8. 2024Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification13 Mar
  9. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  10. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  11. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov

Other references · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Gemma 3 Technical Reportcited by 1,686
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
  6. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  7. 2025OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observationcited by 20
  8. 2024Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagerycited by 14
  9. 2025SARChat-Bench-2M: A Multi-Task Vision-Language Benchmark for SAR Image Interpretationcited by 5
  10. 2025Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Modelcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RSVLM-QA: A Benchmark Dataset for Remote Sensing Vision Language Model-based Question Answering
  2. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
  3. 2024Large Vision-Language Models for Remote Sensing Visual Question Answering
  4. 2026ReVA: A Scene-Centric Dataset Beyond Repetition for Remote Sensing Video Question Answering
  5. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  6. 2026GeoNLI - A Natural Language Interpreter for Satellite Imagery
  7. 2026Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing
  8. 2026Selective Tool Use for Agentic Change Visual Question Answering in Remote Sensing