mlsanityData to 13 Oct 2026

MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment

arXiv 2407.21654 ↗First posted 31 Jul 2024cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent approaches have shown that large-scale vision-language models such as CLIP can improve semantic segmentation performance. These methods typically aim for pixel-level vision-language alignment, but often rely on low resolution image features from CLIP, resulting in class ambiguities along boundaries. Moreover, the global scene representations in CLIP text embeddings do not directly correlate with the local and detailed pixel-level features, making meaningful alignment more difficult. To address these limitations, we introduce MTA-CLIP, a novel framework employing mask-level vision-language alignment. Specifically, we first propose Mask-Text Decoder that enhances the mask representations using rich textual data with the CLIP language model. Subsequently, it aligns mask representations with text embeddings using Mask-to-Text Contrastive Learning. Furthermore, we introduce MaskText Prompt Learning, utilizing multiple context-specific prompts for text embeddings to capture diverse class representations across masks. Overall, MTA-CLIP achieves state-of-the-art, surpassing prior works by an average of 2.8% and 1.3% on on standard benchmark datasets, ADE20k and Cityscapes, respectively.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 50 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

45 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Probabilistic Prompt Learning for Dense Prediction3 Apr
  2. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  3. 2022Prompt Distribution Learning6 May
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,849
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,435
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,240
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,216
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,930
26 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,474
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,344
  3. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,418
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,268
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,150
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,068
  8. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  9. 2022Language-driven Semantic Segmentationcited by 715
  10. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  11. 2019Panoptic Feature Pyramid Networkscited by 644
  12. 2021Extract Free Dense Labels from CLIPcited by 582
  13. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  14. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  15. 2019Selective Kernel Networkscited by 403
  16. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 343
  17. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  18. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 217
  19. 2017Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Networkcited by 208
  20. 2019Gated-SCNN: Gated Shape CNNs for Semantic Segmentationcited by 165
  21. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  22. 2020Context Prior for Scene Segmentationcited by 121
  23. 2019Boundary-Aware Feature Propagation for Scene Segmentationcited by 107
  24. 2020SegFix: Model-Agnostic Boundary Refinement for Segmentationcited by 97
  25. 2021Lite-HRNet: A Lightweight High-Resolution Networkcited by 91
  26. 2015Semantic Segmentation with Boundary Neural Fieldscited by 37

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  2. 2025Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentationcited by 23
  3. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  4. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentationcited by 5
  5. 2025Marine Saliency Segmenter: Object-Focused Conditional Diffusion with Region-Level Semantic Knowledge Distillationcited by 2
  6. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  7. 2025Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglementcited by 1
  8. 2025Leveraging Text-Driven Semantic Variation for Robust OOD Segmentationcited by 1
  9. 2026Lorentz Framework for Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
  2. 2026Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling
  3. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  4. 2024Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation
  5. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
  6. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification
  7. 2024High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation
  8. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation