mlsanityData to 7 Oct 2026

LiteEmbed: Adapting CLIP to Rare Classes

arXiv 2601.09661 ↗First posted 14 Jan 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large-scale vision-language models such as CLIP achieve strong zero-shot recognition but struggle with classes that are rarely seen during pretraining, including newly emerging entities and culturally specific categories. We introduce LiteEmbed, a lightweight framework for few-shot personalization of CLIP that enables new classes to be added without retraining its encoders. LiteEmbed performs subspace-guided optimization of text embeddings within CLIP's vocabulary, leveraging a PCA-based decomposition that disentangles coarse semantic directions from fine-grained variations. Two complementary objectives, coarse alignment and fine separation, jointly preserve global semantic consistency while enhancing discriminability among visually similar classes. Once optimized, the embeddings are plug-and-play, seamlessly substituting CLIP's original text features across classification, retrieval, segmentation, and detection tasks. Extensive experiments demonstrate substantial gains over prior methods, establishing LiteEmbed as an effective approach for adapting CLIP to underrepresented, rare, or unseen classes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 47 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

43 of 47 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting26 Mar
  2. 2025Efficient Few-Shot Continual Learning in Vision-Language Models6 Feb
  3. 2025DynaPrompt: Dynamic Test-Time Prompt Tuning27 Jan
  4. 2024Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifier29 Oct
  5. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion21 Mar
  6. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  7. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  8. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  10. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  11. 2022Prompt Distribution Learning6 May
  12. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
2 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  8. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  11. 2021Image Segmentation Using Text and Image Promptscited by 509
  12. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
17 more
  1. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  2. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  3. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  4. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  5. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  6. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  7. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  8. 2022CLIP model is an Efficient Continual Learnercited by 64
  9. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  10. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  11. 2023Turning a CLIP Model into a Scene Text Detectorcited by 44
  12. 2023An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesiscited by 31
  13. 2025External Knowledge Injection for CLIP-Based Class-Incremental Learningcited by 10
  14. 2025Advancing Prompt-Based Methods for Replay-Independent General Continual Learningcited by 9
  15. 2023New Benchmarks for Asian Facial Recognition Tasks: Face Classification with Large Foundation Modelscited by 4
  16. 2024TV100: A TV Series Dataset that Pre-Trained CLIP Has Not Seencited by 2
  17. 2025Descriminative-Generative Custom Tokens for Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP
  2. 2025Compositional Image-Text Matching and Retrieval by Grounding Entities
  3. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification
  4. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  5. 2026Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis
  6. 2023LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models
  7. 2025CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier
  8. 2024Finetuning CLIP to Reason about Pairwise Differences