# HyperCap: Hyperspectral Land Cover Captioning Dataset for Vision Language Models

arXiv: https://arxiv.org/abs/2505.12217 · HTML: https://mlsanity.com/paper/2505.12217/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2103.05569/) (2021)
- [Deep Learning for Hyperspectral Image Classification: An Overview](https://mlsanity.com/paper/1910.12861/) (2019)
- [Deep Learning for Classification of Hyperspectral Data: A Comparative Review](https://mlsanity.com/paper/1904.10674/) (2019)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [A Survey: Deep Learning for Hyperspectral Image Classification with Few Labeled Samples](https://mlsanity.com/paper/2112.01800/) (2021)
- [Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification](https://mlsanity.com/paper/2209.02700/) (2022)
- [Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval](https://mlsanity.com/paper/2310.08276/) (2023)
- [Hyperspectral Image Super Resolution with Real Unaligned RGB Guidance](https://mlsanity.com/paper/2302.06298/) (2023)
- [Deep Learning Techniques for Hyperspectral Image Analysis in Agriculture: A Review](https://mlsanity.com/paper/2304.13880/) (2023)
- [AeroRIT: A New Scene for Hyperspectral Image Analysis](https://mlsanity.com/paper/1912.08178/) (2019)
- [3D-Convolution Guided Spectral-Spatial Transformer for Hyperspectral Image Classification](https://mlsanity.com/paper/2404.13252/) (2024)
- [3D-RCNet: Learning from Transformer to Build a 3D Relational ConvNet for Hyperspectral Image Classification](https://mlsanity.com/paper/2408.13728/) (2024)
- [AI-Driven HSI: Multimodality, Fusion, Challenges, and the Deep Learning Revolution](https://mlsanity.com/paper/2502.06894/) (2025)

## Built on by (most cited first, 3 of 3)

- [RAH-VLA: Resolution-Adaptive Hierarchical Vision-Language Alignment for Multimodal Remote Sensing Understanding](https://mlsanity.com/paper/2512.23243/) (2025)
- [Agentic Multimodal Models for Environmental Hyperspectral Unmixing](https://mlsanity.com/paper/2609.01289/) (2026)
- [Hyperspectral Image Models: Technical Report](https://mlsanity.com/paper/2609.39871/) (2026)

## Abstract (arXiv)

We introduce HyperCap, the first large-scale hyperspectral captioning dataset designed to enhance model performance and effectiveness in remote sensing applications. Unlike traditional hyperspectral imaging (HSI) benchmarks, HyperCap integrates spectral data with pixel-wise textual annotations, enabling deeper semantic understanding. This dataset enhances model performance in tasks like classification and feature extraction, providing a valuable resource for advanced remote sensing applications. HyperCap is constructed from four benchmark datasets and annotated through a hybrid approach combining automated and manual methods to ensure accuracy and consistency. Empirical evaluations using state-of-the-art encoders and diverse fusion techniques demonstrate significant improvements in classification performance. These results underscore the potential of vision-language learning in HSI and position HyperCap as a foundational dataset for future research in the field. The code and dataset are available at https://github.com/arya-domain/HyperCap.