mlsanityData to 14 Oct 2026

Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation

arXiv 2608.29923 ↗First posted 30 Aug 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Open-vocabulary semantic segmentation (OVSS) relies on vision-language alignment to recognize arbitrary text-defined categories, yet this alignment is fragile under continual test-time distribution shift. Our diagnostic analysis reveals that entropy minimization drives patch-level class collapse, continual updates erode vision-language alignment, and redundant gradients from low-shift samples waste computation. We propose Diversify, Anchor, and Filter (DAF), a stabilization framework that augments entropy-based adaptation with a marginal diversity loss that resists collapse, a cross-modal anchor consistency loss that constrains feature drift relative to a frozen source model, and feature salience filtering that skips low-value backward passes to offset part of the source-anchor overhead. We evaluate on five datasets spanning natural scenes, autonomous driving, underwater imagery, and remote sensing with their corrupted variants. Across the evaluated continual shifts, DAF remains stable where entropy minimization collapses, improving mIoU by over 8 points on Pascal VOC20-C, over 9 points on LoveDA, and over 3 points on Foggy Cityscapes compared to the source model, and is robust to aggressive adaptation and learning rate choices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

31 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Training-Free Class Purification for Open-Vocabulary Semantic Segmentation1 Aug
  2. 2025BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis11 Jul
  3. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation28 May
  4. 2025Bayesian Test-Time Adaptation for Vision-Language Models12 Mar
  5. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time1 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  4. 2022Continual Test-Time Domain Adaptationcited by 525
  5. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  6. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  7. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
  8. 2017Semantic Foggy Scene Understanding with Synthetic Datacited by 403
  9. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 261
  10. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  11. 2021Source-Free Domain Adaptation for Semantic Segmentationcited by 168
  12. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
14 more
  1. 2024Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factorscited by 106
  2. 2020Semantic Segmentation of Underwater Imagery: Dataset and Benchmarkcited by 95
  3. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  4. 2021If your data distribution shifts, use self-learningcited by 36
  5. 2024DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptationcited by 33
  6. 2023Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptationcited by 30
  7. 2023Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentationcited by 29
  8. 2022Test-Time Adaptation with Shape Moments for Image Segmentationcited by 19
  9. 2023MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentationcited by 18
  10. 2023Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentationcited by 10
  11. 2019Underwater Fish Detection with Weak Multi-Domain Supervisioncited by 5
  12. 2023Few Clicks Suffice: Active Test-Time Adaptation for Semantic Segmentationcited by 3
  13. 2025Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentationcited by 3
  14. 2026Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  2. 2024Transferable and Principled Efficiency for Open-Vocabulary Segmentation
  3. 2024Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
  4. 2026Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
  5. 2026dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
  6. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
  7. 2025InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
  8. 2025OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentation