# MVT: Mask-Grounded Vision-Language Models for Taxonomy-Aligned Land-Cover Tagging

arXiv: https://arxiv.org/abs/2509.18693 · HTML: https://mlsanity.com/paper/2509.18693/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 38)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2110.08733/) (2021)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval](https://mlsanity.com/paper/1706.03424/) (2017)
- [OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping](https://mlsanity.com/paper/2210.10732/) (2022)
- [Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild](https://mlsanity.com/paper/2203.03800/) (2022)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images](https://mlsanity.com/paper/2104.12137/) (2021)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [Noise-Consistent Siamese-Diffusion for Medical Image Synthesis and Segmentation](https://mlsanity.com/paper/2505.06068/) (2025)
- [Remote Sensing Object Detection Meets Deep Learning: A Meta-review of Challenges and Advances](https://mlsanity.com/paper/2309.06751/) (2023)

## Abstract (arXiv)

Land-cover understanding in remote sensing increasingly demands class-agnostic systems that generalize across datasets while remaining spatially precise and interpretable. We study a geometry-first discovery-and-interpretation setting under domain shift, where candidate regions are delineated class-agnostically and supervision avoids lexical class names via anonymized identifiers. Complementary to open-set recognition and open-world learning, we focus on coupling class-agnostic mask evidence with taxonomy-grounded scene interpretation, rather than unknown rejection or continual class expansion. We propose MVT, a three-stage framework that (i) extracts boundary-faithful region masks using SAM2 with domain adaptation, (ii) performs mask-grounded semantic tagging and scene description generation via dual-step LoRA fine-tuning of multimodal LLMs, and (iii) evaluates outputs with LLM-as-judge scoring calibrated by stratified expert ratings. On cross-dataset segmentation transfer (train on OpenEarthMap, evaluate on LoveDA), domain-adapted SAM2 improves mask quality; meanwhile, dual-step MLLM fine-tuning yields more accurate taxonomy-aligned tags and more informative mask-grounded scene descriptions.