# Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation

arXiv: https://arxiv.org/abs/2501.16467 · HTML: https://mlsanity.com/paper/2501.16467/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 11 of 11)

- [Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models](https://mlsanity.com/paper/2410.19732/) (2024)
- [Towards Robust Ranker for Text Retrieval](https://mlsanity.com/paper/2206.08063/) (2022)
- [DEAL: Difficulty-aware Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2010.08705/) (2020)
- [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022)
- [MemoryMamba: Memory-Augmented State Space Model for Defect Recognition](https://mlsanity.com/paper/2405.03673/) (2024)
- [Improving Cross-modal Alignment for Text-Guided Image Inpainting](https://mlsanity.com/paper/2301.11362/) (2023)
- [Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overview](https://mlsanity.com/paper/2211.08352/) (2022)
- [Learning Panoptic Segmentation from Instance Contours](https://mlsanity.com/paper/2010.11681/) (2020)
- [Robust 3D Scene Segmentation through Hierarchical and Learnable Part-Fusion](https://mlsanity.com/paper/2111.08434/) (2021)
- [Prototype Guided Network for Anomaly Segmentation](https://mlsanity.com/paper/2201.05869/) (2022)
- [Semantic Segmentation by Semantic Proportions](https://mlsanity.com/paper/2305.15608/) (2023)

## Abstract (arXiv)

Semantic segmentation plays a crucial role in enabling machines to understand and interpret visual scenes at a pixel level. While traditional segmentation methods have achieved remarkable success, their generalization to diverse scenes and unseen object categories remains limited. Recent advancements in large language models (LLMs) offer a promising avenue for bridging visual and textual modalities, providing a deeper understanding of semantic relationships. In this paper, we propose LangSeg, a novel LLM-guided semantic segmentation method that leverages context-sensitive, fine-grained subclass descriptors generated by LLMs. Our framework integrates these descriptors with a pre-trained Vision Transformer (ViT) to achieve superior segmentation performance without extensive model retraining. We evaluate LangSeg on two challenging datasets, ADE20K and COCO-Stuff, where it outperforms state-of-the-art models, achieving up to a 6.1% improvement in mean Intersection over Union (mIoU). Additionally, we conduct a comprehensive ablation study and human evaluation to validate the effectiveness of our method in real-world scenarios. The results demonstrate that LangSeg not only excels in semantic understanding and contextual alignment but also provides a flexible and efficient framework for language-guided segmentation tasks. This approach opens up new possibilities for interactive and domain-specific segmentation applications.