mlsanityData to 14 Oct 2026

Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

arXiv 2607.22205 ↗First posted 24 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Remote sensing multimodal large language models (RS-MLLMs) have advanced aerial-image understanding, but scenario specialization remains constrained by scarce high-quality supervision and incomplete capability coverage. We propose filling before advancing (FBA), a capability-gap-driven post-training route that develops prerequisite capabilities before scenario-specific adaptation. For coastal harbor understanding, we construct CPRS (Coastal-Port Remote Sensing), a three-layer dataset covering RGB, SAR, PAN, and NIR imagery. FBA organizes supervision into RS semantic anchoring, multi-source bridging, and evidence-grounded scenario tuning. We also introduce HarborEval, an eight-track benchmark assessing perception, spatial understanding, robustness, and generation. Using the same final-task supervision pool, FBA raises the HarborEval score from 57.95 to 70.29 on LLaVA-v1.5 and from 81.09 to 83.37 on Qwen3-VL compared with Direct-SFT. It also outperforms collapsed training and leads the evaluated comparisons on harbor-related VRSBench and RSVQA subsets and expert-scored OpenEval. Stage-wise and supervision-replacement analyses examine the contributions of the supervision roles. A lightweight urban instantiation further supports cross-scenario applicability, with the proposed stage order achieving stronger overall performance than the tested alternatives. Project resources and release updates are available at https://github.com/Z0ngL1ng/filling-before-advancing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 45 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

31 of 45 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues19 Dec
  2. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation14 Nov
  3. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding18 Jun
  4. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding14 Jun
  5. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis29 Mar
  6. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  7. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  8. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing20 Dec
  9. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  10. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  11. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing20 Jun
  12. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2018Experience Replay for Continual Learningcited by 1,095
  10. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 623
  11. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 340
  12. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
7 more
  1. 2019BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understandingcited by 234
  2. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  3. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  4. 2019SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusioncited by 95
  5. 2020Deep Learning Meets SARcited by 60
  6. 2024ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Modelscited by 24
  7. 2023PanBench: Towards High-Resolution and High-Performance Pansharpeningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  2. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
  3. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation
  4. 2024RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understanding
  5. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
  6. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
  7. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
  8. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding