mlsanityData to 7 Oct 2026

Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning

arXiv 2308.06038 ↗First posted 11 Aug 2023cs.CVcited by 150

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Benefiting from prompt tuning, recent years have witnessed the promising performance of pre-trained vision-language models, e.g., CLIP, on versatile downstream tasks. In this paper, we focus on a particular setting of learning adaptive prompts on the fly for each test sample from an unseen new domain, which is known as test-time prompt tuning (TPT). Existing TPT methods typically rely on data augmentation and confidence selection. However, conventional data augmentation techniques, e.g., random resized crops, suffers from the lack of data diversity, while entropy-based confidence selection alone is not sufficient to guarantee prediction fidelity. To address these issues, we propose a novel TPT method, named DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Specifically, we incorporate augmented data by both conventional method and pre-trained stable diffusion to exploit their respective merits, improving the models ability to adapt to unknown new test data. Moreover, to ensure the prediction fidelity of generated data, we introduce a cosine similarity-based filtration technique to select the generated data with higher similarity to the single test sample. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13% compared to the state-of-the-art TPT method. Our code and models will be publicly released.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 68 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

49 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  2. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
31 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2023Consistency Modelscited by 1,778
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2017Random Erasing Data Augmentationcited by 1,235
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  14. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  15. 2022On Distillation of Guided Diffusion Modelscited by 721
  16. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  17. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  18. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  19. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  20. 2017Data Augmentation Generative Adversarial Networkscited by 278
  21. 2022Contrastive Test-Time Adaptationcited by 258
  22. 2023Token Merging for Fast Stable Diffusioncited by 220
  23. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  24. 2020Better Aggregation in Test-Time Augmentationcited by 121
  25. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  26. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  27. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  28. 2019Out-of-Distribution Detection for Generalized Zero-Shot Action Recognitioncited by 54
  29. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  30. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  31. 2022Prompt-driven efficient Open-set Semi-supervised Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 40; 2025: 69; 2026: 38.

150 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  3. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  4. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  5. 2023Sentence-level Prompts Benefit Composed Image Retrievalcited by 59
  6. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  7. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  8. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  9. 2024Boosting Vision-Language Models with Transductioncited by 36
  10. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  11. 2024WATT: Weight Average Test-Time Adaptation of CLIPcited by 33
  12. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
38 more
  1. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 31
  2. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  3. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  4. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  5. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Modelscited by 22
  6. 2024Image Generation Diversity Issues and How to Tame Themcited by 22
  7. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  8. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  9. 2024Distribution-Aware Data Expansion with Diffusion Modelscited by 20
  10. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  11. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Modelscited by 17
  12. 2024BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrappingcited by 17
  13. 2025From Informal to Formal -- Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofscited by 17
  14. 2024DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Datacited by 15
  15. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
  16. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  17. 2024Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Modelscited by 13
  18. 2025COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptationcited by 13
  19. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  20. 2024Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentationcited by 12
  21. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 12
  22. 2025SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Promptingcited by 11
  23. 2024A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Modelscited by 10
  24. 2024Generative Active Learning for Long-tailed Instance Segmentationcited by 10
  25. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  26. 2024Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Modelscited by 9
  27. 2024Efficient Test-Time Prompt Tuning for Vision-Language Modelscited by 9
  28. 2025Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discoverycited by 9
  29. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variationscited by 9
  30. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  31. 2023GeNIe: Generative Hard Negative Images Through Diffusioncited by 8
  32. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 8
  33. 2024Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Modelcited by 8
  34. 2025Noisy Test-Time Adaptation in Vision-Language Modelscited by 8
  35. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  36. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 8
  37. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  38. 2025Test-Time Visual In-Context Tuningcited by 7

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
  2. 2026USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning
  3. 2025DynaPrompt: Dynamic Test-Time Prompt Tuning
  4. 2025Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
  5. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learning
  6. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
  7. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  8. 2024Generating Synthetic Datasets for Few-shot Prompt Tuning