# Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning

arXiv: https://arxiv.org/abs/2604.12403 · HTML: https://mlsanity.com/paper/2604.12403/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning](https://mlsanity.com/paper/2308.06038/) (2023)
- [Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors](https://mlsanity.com/paper/2403.07366/) (2024)
- [Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization](https://mlsanity.com/paper/2311.01459/) (2023)
- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [Consistency-guided Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2306.01195/) (2023)
- [I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2212.02291/) (2022)
- [Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts](https://mlsanity.com/paper/2307.11661/) (2023)
- [Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2211.02219/) (2022)
- [A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models](https://mlsanity.com/paper/2302.06235/) (2023)
- [DynaPrompt: Dynamic Test-Time Prompt Tuning](https://mlsanity.com/paper/2501.16404/) (2025)
- [AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation](https://mlsanity.com/paper/2407.04603/) (2024)
- [Any-Shift Prompting for Generalization over Distributions](https://mlsanity.com/paper/2402.10099/) (2024)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)

## Built on by (most cited first, 2 of 2)

- [Rethinking the Test-Time Prompt Tuning Objective from the Perspective of Calibration](https://mlsanity.com/paper/2608.30230/) (2026)
- [Balanced Prompt Adaptation against Entropy-Induced Collapse for Test-Time Binary Segmentation](https://mlsanity.com/paper/2609.21743/) (2026)

## Abstract (arXiv)

Test-Time Prompt Tuning (TPT) adapts vision-language models using augmented views, but its effectiveness is hindered by the challenge of determining which views are beneficial. Standard entropy-based filtering relies on the internal confidence scores of the model, which are often miscalibrated under distribution shift, assigning high confidence to irrelevant crops or background regions while ignoring semantic content. To address this, we propose a dual-modality anchor-guided framework that grounds view selection in semantic evidence. We introduce a text anchor from attribute-rich descriptions, to provide fine-grained class semantics, and an adaptive image anchor that captures evolving test-time statistics. Using these anchors, we filter views based on alignment and confidence, ensuring that only informative views guide adaptation. Moreover, we treat the anchors as auxiliary predictive heads and combine their predictions with the original output in a confidence-weighted ensemble, yielding a stable supervision signal for prompt updates. Extensive experiments on 15 benchmark datasets demonstrate new state-of-the-art performance, highlighting the contribution of anchor-guided supervision as a foundation for robust prompt updates.