mlsanityData to 7 Oct 2026

Textual Query-Driven Mask Transformer for Domain Generalized Segmentation

arXiv 2407.09033 ↗First posted 12 Jul 2024cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In this paper, we introduce a method to tackle Domain Generalized Semantic Segmentation (DGSS) by utilizing domain-invariant semantic knowledge from text embeddings of vision-language models. We employ the text embeddings as object queries within a transformer-based segmentation framework (textual object queries). These queries are regarded as a domain-invariant basis for pixel grouping in DGSS. To leverage the power of textual object queries, we introduce a novel framework named the textual query-driven mask transformer (tqdm). Our tqdm aims to (1) generate textual object queries that maximally encode domain-invariant semantics and (2) enhance the semantic clarity of dense visual features. Additionally, we suggest three regularization losses to improve the efficacy of tqdm by aligning between visual and textual features. By utilizing our method, the model can comprehend inherent semantic information for classes of interest, enabling it to generalize to extreme domains (e.g., sketch style). Our tqdm achieves 68.9 mIoU on GTA5rightarrowCityscapes, outperforming the prior state-of-the-art method by 2.5 mIoU. The project page is available at https://byeonghyunpak.github.io/tqdm.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 72 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

68 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Strong but simple: A Baseline for Domain Generalized Dense Perception by CLIP-based Transfer Learning4 Dec
  2. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance21 Sep
  3. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training12 Apr
  4. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
38 more
  1. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  6. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2022Language-driven Semantic Segmentationcited by 711
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  14. 2021Extract Free Dense Labels from CLIPcited by 580
  15. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  16. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  18. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  19. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  20. 2018Two at Once: Enhancing Learning and Generalization Capacities via IBN-Netcited by 361
  21. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  22. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  23. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 245
  24. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  25. 2021RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whiteningcited by 213
  26. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  27. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  28. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  29. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  30. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  31. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  32. 2019Switchable Whitening for Deep Representation Learningcited by 105
  33. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  34. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  35. 2022Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentationcited by 88
  36. 2023Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 82
  37. 2022Semantic-Aware Domain Generalized Segmentationcited by 78
  38. 2022WildNet: Learning Domain Generalized Semantic Segmentation from the Wildcited by 75

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentationcited by 23
  2. 2024SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learningcited by 22
  3. 2025FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentationcited by 15
  4. 2025Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentationcited by 4
  5. 2025Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Modelscited by 2
  6. 2025Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 2
  7. 2026Open-Vocabulary Domain Generalization in Urban-Scene Segmentationcited by 2
  8. 2025Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentationcited by 1
  9. 2025Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentationcited by 1
  10. 2025Leveraging Text-Driven Semantic Variation for Robust OOD Segmentationcited by 1
  11. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  12. 2025Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Modelscited by 0
8 more
  1. 2025Multi-Granularity Feature Calibration via VFM for Domain Generalized Semantic Segmentationcited by 0
  2. 2025IELDG: Suppressing Domain-Specific Noise with Inverse Evolution Layers for Domain Generalized Semantic Segmentationcited by 0
  3. 2025Vision-Language Semantic Aggregation Leveraging Foundation Model for Generalizable Medical Image Segmentationcited by 0
  4. 2025Domain Generalization for Semantic Segmentation: A Surveycited by 0
  5. 2025OMUDA: Omni-level Masking for Unsupervised Domain Adaptation in Semantic Segmentationcited by 0
  6. 2026OT-Drive: Out-of-Distribution Off-Road Traversable Area Segmentation via Optimal Transportcited by 0
  7. 2026RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentationcited by 0
  8. 2026Hierarchical Prompt Injector for Domain Generalization Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LASA: Language-and-Source-Anchored Alignment for Domain Generalized Semantic Segmentation
  2. 2024Text4Seg: Reimagining Image Segmentation as Text Generation
  3. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modeling
  4. 2023Generalization by Adaptation: Diffusion-Based Domain Extension for Domain-Generalized Semantic Segmentation
  5. 2026Toward Semantic-Agnostic and Shape-Aware Vision-Language Segmentation Models
  6. 2024Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
  7. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
  8. 2025Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval