Exploiting LMM-based knowledge for image classification tasks
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
In this paper we address image classification tasks leveraging knowledge encoded in Large Multimodal Models (LMMs). More specifically, we use the MiniGPT-4 model to extract semantic descriptions for the images, in a multimodal prompting fashion. In the current literature, vision language models such as CLIP, among other approaches, are utilized as feature extractors, using only the image encoder, for solving image classification tasks. In this paper, we propose to additionally use the text encoder to obtain the text embeddings corresponding to the MiniGPT-4-generated semantic descriptions. Thus, we use both the image and text embeddings for solving the image classification task. The experimental evaluation on three datasets validates the improved classification performance achieved by exploiting LMM-based knowledge.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 26 references that are in the map. First posted on arXiv 5 Jun 2024.
Builds on
22 of 26 references.
References · 22
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023A Survey of Large Language Modelscited by 2,537
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
- 2020Concept Bottleneck Modelscited by 1,022
- 2023A Survey on Multimodal Large Language Modelscited by 898
- 2014Show and Tell: A Neural Image Caption Generatorcited by 877
- 2023A Comprehensive Overview of Large Language Modelscited by 639
10 more
- 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
- 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
- 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
- 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
- 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Promptscited by 48
- 2023Turning a CLIP Model into a Scene Text Detectorcited by 44
- 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
- 2023SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Modelcited by 18
- 2023PatFig: Generating Short and Long Captions for Patent Figurescited by 12
Built on by
Citing papers per year: 2024: 2.
2 papers.
- 2024LMM-Regularized CLIP Embeddings for Image Classification16 Dec 2024
- 2024Disturbing Image Detection Using LMM-Elicited Emotion Embeddings18 Jun 2024
Similar, not cited
Close in wording, no citation either way
- 2025On Large Multimodal Models as Open-World Image Classifiers
- 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
- 2023Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding
- 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
- 2025Large Language Models Facilitate Vision Reflection in Image Classification
- 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
- 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
- 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features