# SonarLLM: A Native Sonar--Optical Multimodal Large Language Model for Underwater Perception

arXiv: https://arxiv.org/abs/2608.24325 · HTML: https://mlsanity.com/paper/2608.24325/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe](https://mlsanity.com/paper/2509.18154/) (2025)
- [Fast Underwater Image Enhancement for Improved Visual Perception](https://mlsanity.com/paper/1903.09766/) (2019)
- [MarineGPT: Unlocking Secrets of Ocean to the Public](https://mlsanity.com/paper/2310.13596/) (2023)
- [Resilient Sensor Fusion under Adverse Sensor Failures via Multi-Modal Expert Fusion](https://mlsanity.com/paper/2503.19776/) (2025)
- [AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysis](https://mlsanity.com/paper/2502.01785/) (2025)
- [RGB-Sonar Tracking Benchmark and Spatial Cross-Attention Transformer Tracker](https://mlsanity.com/paper/2406.07189/) (2024)
- [NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding](https://mlsanity.com/paper/2510.27481/) (2025)
- [A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception](https://mlsanity.com/paper/2606.01398/) (2026)

## Abstract (arXiv)

Reliable underwater perception requires complementary sensing under variable visibility. Optical cameras capture appearance and semantics but degrade rapidly with turbidity, whereas imaging sonar preserves geometry while exhibiting distinct range-azimuth structure and acoustic artifacts. Existing MLLMs, built primarily on optical encoders, are therefore ill-suited to model sonar or adaptively exploit sonar-optical complementarity. We propose SonarLLM, a sonar-optical MLLM that treats sonar as a native perceptual modality. It combines a sonar-specific encoder, modality-specific physics-aware feature enhancement, and reliability-aware hierarchical fusion to align acoustic structure with optical semantics and dynamically adjust their contributions as sensing quality changes. We also introduce SonarBench, a paired benchmark that spans four tasks: recognition, counting, visual question answering, and captioning; and, across the benchmark, three input settings: sonar-only, optical-only, and fusion. By fixing the scene and sonar observation while varying optical degradation, SonarBench enables controlled measurement of cross-modal complementarity. SonarLLM achieves 72.0% macro accuracy across sonar-only recognition, counting, and VQA, outperforming the strongest baseline by 34.4 percentage points, and 68.7% under fusion, exceeding the best baseline by 25.1 points. For recognition and counting, the fusion-over-optical gain grows from 6.0 to 36.0 points as turbidity increases, indicating the increasing complementary value of sonar under controlled optical degradation. Together, these results show that robust heterogeneous perception depends not only on adding sonar, but on representing and weighting it according to its sensing characteristics.