# Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation

arXiv: https://arxiv.org/abs/2405.18840 · HTML: https://mlsanity.com/paper/2405.18840/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2024.

## Built on by (most cited first, 8 of 8)

- [Learning to Instruct for Visual Instruction Tuning](https://mlsanity.com/paper/2503.22215/) (2025)
- [PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation](https://mlsanity.com/paper/2603.17520/) (2026)
- [Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge](https://mlsanity.com/paper/2510.08316/) (2025)
- [TALENT: Target-aware Efficient Tuning for Referring Image Segmentation](https://mlsanity.com/paper/2604.00609/) (2026)
- [Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2605.08874/) (2026)
- [AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition](https://mlsanity.com/paper/2508.11870/) (2025)
- [dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3](https://mlsanity.com/paper/2603.19531/) (2026)
- [Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2604.26221/) (2026)

## Abstract (arXiv)

Open-vocabulary semantic segmentation seeks to label each pixel in an image with arbitrary text descriptions. Vision-language foundation models, especially CLIP, have recently emerged as powerful tools for acquiring open-vocabulary capabilities. However, fine-tuning CLIP to equip it with pixel-level prediction ability often suffers three issues: 1) high computational cost, 2) misalignment between the two inherent modalities of CLIP, and 3) degraded generalization ability on unseen categories. To address these issues, we propose H-CLIP a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both of the two CLIP modalities. Specifically, the PEFT strategy is achieved by a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices. Since the PEFT strategy is conducted symmetrically to the two CLIP modalities, the misalignment between them is mitigated. Furthermore, we apply an additional constraint to PEFT on the CLIP text encoder according to the hyperspherical energy principle, i.e., minimizing hyperspherical energy during fine-tuning preserves the intrinsic structure of the original parameter space, to prevent the destruction of the generalization ability offered by the CLIP text encoder. Extensive evaluations across various benchmarks show that H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4% of the total parameters of CLIP.