# LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System

arXiv: https://arxiv.org/abs/2601.13096 · HTML: https://mlsanity.com/paper/2601.13096/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community](https://mlsanity.com/paper/2408.09110/) (2024)
- [Predictive Modeling of Flexible EHD Pumps using Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.07488/) (2024)
- [Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagery](https://mlsanity.com/paper/2403.03790/) (2024)
- [OceanPlan: Hierarchical Planning and Replanning for Natural Language AUV Piloting in Large-scale Unexplored Ocean Environments](https://mlsanity.com/paper/2403.15369/) (2024)
- [Word2Wave: Language Driven Mission Programming for Efficient Subsea Deployments of Marine Robots](https://mlsanity.com/paper/2409.18405/) (2024)
- [Large Language Model-based Decision-making for COLREGs and the Control of Autonomous Surface Vehicles](https://mlsanity.com/paper/2411.16587/) (2024)
- [Maritime Mission Planning for Unmanned Surface Vessel using Large Language Model](https://mlsanity.com/paper/2503.12065/) (2025)
- [Collaborative Object Manipulation on the Water Surface by a UAV-USV Team Using Tethers](https://mlsanity.com/paper/2407.08580/) (2024)
- [Towards UAV-USV Collaboration in Harsh Maritime Conditions Including Large Waves](https://mlsanity.com/paper/2408.10163/) (2024)
- [Long-Range Vision-Based UAV-assisted Localization for Unmanned Surface Vehicles](https://mlsanity.com/paper/2408.11429/) (2024)
- [Model predictive control-based trajectory generation for agile landing of unmanned aerial vehicle on a moving boat](https://mlsanity.com/paper/2412.07332/) (2024)
- [AquaChat: An LLM-Guided ROV Framework for Adaptive Inspection of Aquaculture Net Pens](https://mlsanity.com/paper/2507.16841/) (2025)
- [Design, Integration and Sea Trials of 3D Printed Unmanned Aerial Vehicle and Unmanned Surface Vehicle for Cooperative Missions](https://mlsanity.com/paper/2102.10709/) (2021)
- [A Search Strategy and Vessel Detection in Maritime Environment Using Fixed-Wing UAVs](https://mlsanity.com/paper/2306.12767/) (2023)
- [A Time and Place to Land: Online Learning-Based Distributed MPC for Multirotor Landing on Surface Vessel in Waves](https://mlsanity.com/paper/2410.21674/) (2024)
- [Drone Carrier: An Integrated Unmanned Surface Vehicle for Autonomous Inspection and Intervention in GNSS-Denied Maritime Environment](https://mlsanity.com/paper/2501.12869/) (2025)
- [A Review of Generative AI in Aquaculture: Foundations, Applications, and Future Directions for Smart and Sustainable Farming](https://mlsanity.com/paper/2507.11974/) (2025)

## Abstract (arXiv)

Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large Language Models (LLMs) and Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection