# Lightweight Multimodal Adaptation of Vision Language Models for Species Recognition and Habitat Context Interpretation in Drone Thermal Imagery

arXiv: https://arxiv.org/abs/2604.06124 · HTML: https://mlsanity.com/paper/2604.06124/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models](https://mlsanity.com/paper/2401.09083/) (2024)
- [Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning](https://mlsanity.com/paper/2312.01191/) (2023)
- [Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives](https://mlsanity.com/paper/2505.14361/) (2025)
- [IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark](https://mlsanity.com/paper/2507.14449/) (2025)
- [ALSS-YOLO: An Adaptive Lightweight Channel Split and Shuffling Network for TIR Wildlife Detection in UAV Imagery](https://mlsanity.com/paper/2409.06259/) (2024)
- [Open-Vocabulary Object Detection in UAV Imagery: A Review and Future Perspectives](https://mlsanity.com/paper/2507.13359/) (2025)
- [Edge-Optimized Multimodal Learning for UAV Video Understanding via BLIP-2](https://mlsanity.com/paper/2601.08408/) (2026)

## Abstract (arXiv)

This study proposes a lightweight multimodal adaptation framework to bridge the representation gap between RGB-pretrained VLMs and thermal infrared imagery, and demonstrates its practical utility using a real drone-collected dataset. A thermal dataset was developed from drone-collected imagery and was used to fine-tune VLMs through multimodal projector alignment, enabling the transfer of information from RGB-based visual representations to thermal radiometric inputs. Three representative models, including InternVL3-8B-Instruct, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct, were benchmarked under both closed-set and open-set prompting conditions for species recognition and instance enumeration. Among the tested models, Qwen3-VL-8B-Instruct with open-set prompting achieved the best overall performance, with F1 scores of 0.935 for deer, 0.915 for rhino, and 0.968 for elephant, and within-1 enumeration accuracies of 0.779, 0.982, and 1.000, respectively. In addition, combining thermal imagery with simultaneously collected RGB imagery enabled the model to generate habitat-context information, including land-cover characteristics, key landscape features, and visible human disturbance. Overall, the findings demonstrate that lightweight projector-based adaptation provides an effective and practical route for transferring RGB-pretrained VLMs to thermal drone imagery, expanding their utility from object-level recognition to habitat-context interpretation in ecological monitoring.