mlsanityData to 7 Oct 2026

Enhancing Target-unspecific Tasks through a Features Matrix

arXiv 2505.03414 ↗First posted 6 May 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent developments in prompt learning of large Vision-Language Models (VLMs) have significantly improved performance in target-specific tasks. However, these prompting methods often struggle to tackle the target-unspecific or generalizable tasks effectively. It may be attributed to the fact that overfitting training causes the model to forget its general knowledge. The general knowledge has a strong promotion on target-unspecific tasks. To alleviate this issue, we propose a novel Features Matrix (FM) approach designed to enhance these models on target-unspecific tasks. Our method extracts and leverages general knowledge, shaping a Features Matrix (FM). Specifically, the FM captures the semantics of diverse inputs from a deep and fine perspective, preserving essential general knowledge, which mitigates the risk of overfitting. Representative evaluations demonstrate that: 1) the FM is compatible with existing frameworks as a generic and flexible module, and 2) the FM significantly showcases its effectiveness in enhancing target-unspecific tasks (base-to-novel generalization, domain generalization, and cross-dataset generalization), achieving state-of-the-art performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 74 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

47 of 74 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Generalizable Prompt Learning of CLIP: A Brief Overview3 Mar
  2. 2025A Similarity Paradigm Through Textual Regularization Without Forgetting20 Feb
  3. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Models2 Dec
  4. 2024Advancing Prompt Learning through an External Layer29 Jul
  5. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models5 Jun
  6. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  7. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  8. 2022Prompt Distribution Learning6 May
  9. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  10. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  11. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
24 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  4. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  5. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  6. 2021Class-agnostic Object Detection with Multi-modal Transformercited by 85
  7. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  8. 2021Learning with Multiclass AUC: Theory and Algorithmscited by 35
  9. 2022OpenAUC: Towards AUC-Oriented Open-Set Recognitioncited by 33
  10. 2024DVLO: Deep Visual-LiDAR Odometry with Local-to-Global Feature Fusion and Bi-Directional Structure Alignmentcited by 25
  11. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environmentscited by 21
  12. 2024FLAME: Learning to Navigate with Multimodal LLM in Urban Environmentscited by 19
  13. 2024The Why, When, and How to Use Active Learning in Large-Data-Driven 3D Object Detection for Safe Autonomous Driving: An Empirical Explorationcited by 17
  14. 2022Optimizing Two-way Partial AUC with an End-to-end Frameworkcited by 15
  15. 2023ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videoscited by 15
  16. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  17. 2024Progress and Opportunities of Foundation Models in Bioinformaticscited by 13
  18. 2024EMIE-MAP: Large-Scale Road Surface Reconstruction Based on Explicit Mesh and Implicit Encodingcited by 13
  19. 2024PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editingcited by 9
  20. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  21. 2024RankAdaptor: Hierarchical Rank Allocation for Efficient Fine-Tuning Pruned LLMs via Performance Modelcited by 7
  22. 2025SparseFormer: Detecting Objects in HRW Shots via Sparse Vision Transformercited by 7
  23. 2025CAMeL: Cross-modality Adaptive Meta-Learning for Text-based Person Retrievalcited by 6
  24. 2022Predict the Rover Mobility over Soft Terrain using Articulated Wheeled Bevametercited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Deeply-Conditioned Image Compression via Self-Generated Priorscited by 1
  2. 2025Few-Shot Remote Sensing Image Scene Classification with CLIP and Prompt Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability
  2. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  3. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
  4. 2024Revisiting Prompt Pretraining of Vision-Language Models
  5. 2024LPT: Less-overfitting Prompt Tuning for Vision-Language Model
  6. 2024Generalizable Prompt Tuning for Vision-Language Models
  7. 2024Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
  8. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models