mlsanityData to 7 Oct 2026

LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

arXiv 2411.09301 ↗First posted 14 Nov 2024cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Automatically and rapidly understanding Earth's surface is fundamental to our grasp of the living environment and informed decision-making. This underscores the need for a unified system with comprehensive capabilities in analyzing Earth's surface to address a wide range of human needs. The emergence of multimodal large language models (MLLMs) has great potential in boosting the efficiency and convenience of intelligent Earth observation. These models can engage in human-like conversations, serve as unified platforms for understanding images, follow diverse instructions, and provide insightful feedbacks. In this study, we introduce LHRS-Bot-Nova, an MLLM specialized in understanding remote sensing (RS) images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions. LHRS-Bot-Nova features an enhanced vision encoder and a novel bridge layer, enabling efficient visual compression and better language-vision alignment. To further enhance RS-oriented vision-language alignment, we propose a large-scale RS image-caption dataset, generated through feature-guided image recaptioning. Additionally, we introduce an instruction dataset specifically designed to improve spatial recognition abilities. Extensive experiments demonstrate superior performance of LHRS-Bot-Nova across various RS image understanding tasks. We also evaluate different MLLM performances in complex RS perception and instruction following using a complicated multi-choice question evaluation benchmark, providing a reliable guide for future model selection and improvement. Data, code, and models will be available at https://github.com/NJU-LHRS/LHRS-Bot.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 96 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

41 of 96 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding14 Jun
  2. 2024Towards Vision-Language Geo-Foundation Model: A Survey13 Jun
  3. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis29 Mar
  4. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  5. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data31 Jan
  6. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  7. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  8. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing20 Jun

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2024Mixtral of Expertscited by 1,785
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
21 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  9. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  10. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  11. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  12. 2023Demystifying CLIP Datacited by 265
  13. 2024Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observationcited by 97
  14. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  15. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  16. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  17. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 36
  18. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  19. 2024Multiple-Choice Questions are Efficient and Robust LLM Evaluatorscited by 28
  20. 2022METER-ML: A Multi-Sensor Earth Observation Benchmark for Automated Methane Source Mappingcited by 17
  21. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  2. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 19
  3. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 13
  4. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  5. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  6. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  7. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
  8. 2025TinyRS-R1: Compact Multimodal Language Model for Remote Sensingcited by 7
  9. 2025Few-Shot Vision-Language Reasoning for Satellite Imagery via Verifiable Rewardscited by 3
  10. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learningcited by 3
  11. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Taskscited by 3
  12. 2025Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modelingcited by 2
13 more
  1. 2026OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMscited by 2
  2. 2025Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captionscited by 1
  3. 2025FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understandingcited by 1
  4. 2026RSHallu: Dual-Mode Hallucination Evaluation for Remote-Sensing Multimodal Large Language Models with Domain-Tailored Mitigationcited by 1
  5. 2026Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understandingcited by 1
  6. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 1
  7. 2026Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensingcited by 1
  8. 2026YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perceptioncited by 0
  9. 2026SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Modelscited by 0
  10. 2026ForestHG-Trace: Traceable Long-Horizon Ecological Reasoning over Large-Scale Forest Scenescited by 0
  11. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipecited by 0
  12. 2026Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMscited by 0
  13. 2026WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagerycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  2. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  3. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation
  4. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
  5. 2024RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
  6. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
  7. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  8. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks