mlsanityData to 7 Oct 2026

SpecPL: Disentangling Spectral Granularity for Prompt Learning

arXiv 2605.04504 ↗First posted 6 May 2026cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing prompt learning for VLMs exhibits a modality asymmetry, predominantly optimizing text tokens while still relying on frozen visual encoder as holistic extractor and neglecting the spectral granularity essential for fine-grained discrimination. To bridge this, we introduce Disentangling Spectral Granularity for Prompt Learning (SpecPL), which approaches prompt learning from a novel spectral perspective via Counterfactual Granule Supervision. Specifically, we leverage a frozen VAE to decompose visual signals into semantic low-frequency bands and granular high-frequency details. A frozen Visual Semantic Bank anchors text representations to universal low-frequency invariants, mitigating overfitting. Crucially, fine-grained discrimination is driven by counterfactual granule training: by permuting high-frequency signals, we compel the model to explicitly distinguish visual granularity from semantic invariance. Uniquely, SpecPL serves as a universal plug-and-play booster, revitalizing text-oriented baselines like CoOp and MaPLe via visual-side guidance. Experiments on 11 benchmarks demonstrate competitive state-of-the-art performance, achieving a new performance ceiling of 81.51% harmonic-mean accuracy. These results validate that spectral disentanglement with counterfactual supervision effectively bridges the gap in the stability-generalization trade-off. Code is released at https://github.com/Mlrac1e/SpecPL-Prompt-Learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

35 of 44 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
23 more
  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  6. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  7. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  8. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  9. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  10. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  11. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  12. 2022Prompt Distribution Learningcited by 235
  13. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  14. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Modelcited by 75
  15. 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
  16. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  17. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  18. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  19. 2023Large Language Models are Good Prompt Learners for Low-Shot Image Classificationcited by 21
  20. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  21. 2025OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuningcited by 12
  22. 2025Hierarchical Cross-modal Prompt Learning for Vision-Language Modelscited by 12
  23. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
  2. 2026U-shaped Multi-granularity Learning for Vision-Language Models
  3. 2026HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models
  4. 2025A Retrospect to Multi-prompt Learning across Vision and Language
  5. 2026Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning
  6. 2024Revisiting Prompt Pretraining of Vision-Language Models
  7. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  8. 2025Decouple before Align: Visual Disentanglement Enhances Prompt Tuning