mlsanityData to 7 Oct 2026

Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models

arXiv 2401.00127 ↗First posted 30 Dec 2023cs.CV cs.SIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The synergy of language and vision models has given rise to Large Language and Vision Assistant models (LLVAs), designed to engage users in rich conversational experiences intertwined with image-based queries. These comprehensive multimodal models seamlessly integrate vision encoders with Large Language Models (LLMs), expanding their applications in general-purpose language and visual comprehension. The advent of Large Multimodal Models (LMMs) heralds a new era in Artificial Intelligence (AI) assistance, extending the horizons of AI utilization. This paper takes a unique perspective on LMMs, exploring their efficacy in performing image classification tasks using tailored prompts designed for specific datasets. We also investigate the LLVAs zero-shot learning capabilities. Our study includes a benchmarking analysis across four diverse datasets: MNIST, Cats Vs. Dogs, Hymnoptera (Ants Vs. Bees), and an unconventional dataset comprising Pox Vs. Non-Pox skin images. The results of our experiments demonstrate the model's remarkable performance, achieving classification accuracies of 85%, 100%, 77%, and 79% for the respective datasets without any fine-tuning. To bolster our analysis, we assess the model's performance post fine-tuning for specific tasks. In one instance, fine-tuning is conducted over a dataset comprising images of faces of children with and without autism. Prior to fine-tuning, the model demonstrated a test accuracy of 55%, which significantly improved to 83% post fine-tuning. These results, coupled with our prior findings, underscore the transformative potential of LLVAs and their versatile applications in real-world scenarios.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 5 of its 15 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

5 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 4

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  3. 2022Image Data collection and implementation of deep learning-based model in detecting Monkeypox disease using modified VGG16cited by 13
  4. 2023Building Multimodal AI Chatbotscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmarkcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
  2. 2025On Large Multimodal Models as Open-World Image Classifiers
  3. 2024NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?
  4. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features
  5. 2026Large Multimodal Models as General In-Context Classifiers
  6. 2023On the Performance of Multimodal Language Models
  7. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models
  8. 2026Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners