mlsanityData to 14 Oct 2026

LMM-Regularized CLIP Embeddings for Image Classification

arXiv 2412.11663 ↗First posted 16 Dec 2024cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In this paper we deal with image classification tasks using the powerful CLIP vision-language model. Our goal is to advance the classification performance using the CLIP's image encoder, by proposing a novel Large Multimodal Model (LMM) based regularization method. The proposed method uses an LMM to extract semantic descriptions for the images of the dataset. Then, it uses the CLIP's text encoder, frozen, in order to obtain the corresponding text embeddings and compute the mean semantic class descriptions. Subsequently, we adapt the CLIP's image encoder by adding a classification head, and we train it along with the image encoder output, apart from the main classification objective, with an additional auxiliary objective. The additional objective forces the embeddings at the image encoder's output to become similar to their corresponding LMM-generated mean semantic class descriptions. In this way, it produces embeddings with enhanced discrimination ability, leading to improved classification performance. The effectiveness of the proposed regularization method is validated through extensive experiments on three image classification datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 19 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

14 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts21 Jul
  2. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2023A Survey on Multimodal Large Language Modelscited by 909
  7. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  8. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  9. 2023Delving into CLIP latent space for Video Anomaly Recognitioncited by 27
  10. 2024Disturbing Image Detection Using LMM-Elicited Emotion Embeddingscited by 4
  11. 2024Exploiting LMM-based knowledge for image classification taskscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TLAC: Two-stage LMM Augmented CLIP for Zero-Shot Classification
  2. 2024LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts
  3. 2024Does VLM Classification Benefit from LLM Description Semantics?
  4. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
  5. 2025On Large Multimodal Models as Open-World Image Classifiers
  6. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
  7. 2026Large Multimodal Models as General In-Context Classifiers
  8. 2024LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation