mlsanityData to 14 Oct 2026

Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs

arXiv 2403.11755 ↗First posted 18 Mar 2024cs.CV cs.AI cs.LGcited by 29

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt ensembling of Large Language Model (LLM) generated category-specific prompts has emerged as an effective method to enhance zero-shot recognition ability of Vision-Language Models (VLMs). To obtain these category-specific prompts, the present methods rely on hand-crafting the prompts to the LLMs for generating VLM prompts for the downstream tasks. However, this requires manually composing these task-specific prompts and still, they might not cover the diverse set of visual concepts and task-specific styles associated with the categories of interest. To effectively take humans out of the loop and completely automate the prompt generation process for zero-shot recognition, we propose Meta-Prompting for Visual Recognition (MPVR). Taking as input only minimal information about the target task, in the form of its short natural language description, and a list of associated class labels, MPVR automatically produces a diverse set of category-specific prompts resulting in a strong zero-shot classifier. MPVR generalizes effectively across various popular zero-shot image recognition benchmarks belonging to widely different domains when tested with multiple LLMs and VLMs. For example, MPVR obtains a zero-shot recognition improvement over CLIP by up to 19.8% and 18.2% (5.0% and 4.5% on average over 20 datasets) leveraging GPT and Mixtral LLMs, respectively

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 57 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

42 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
  2. 2022Visual Classification via Description from Large Language Models13 Oct
  3. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
25 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2013Describing Textures in the Wildcited by 2,125
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2024Mixtral of Expertscited by 1,823
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  13. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  14. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  15. 2021MetaICL: Learning to Learn In Contextcited by 454
  16. 2023Demystifying CLIP Datacited by 270
  17. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  18. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  19. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  20. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  21. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
  22. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  23. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  24. 2022MetaPrompting: Learning to Learn Better Promptscited by 19
  25. 2023TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classificationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 11; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  2. 2024Beyond Model Adaptation at Test Time: A Surveycited by 59
  3. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  4. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  5. 2024IPO: Interpretable Prompt Optimization for Vision-Language Modelscited by 14
  6. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  7. 2025ProAPO: Progressively Automatic Prompt Optimization for Visual Classificationcited by 13
  8. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  9. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  10. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Modelscited by 12
  11. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Modelscited by 10
  12. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 10
17 more
  1. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  2. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  3. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  4. 2024Proactive Schemes: A Survey of Adversarial Attacks for Social Goodcited by 2
  5. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  6. 2025MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classificationcited by 2
  7. 2025Draft and Refine with Visual Expertscited by 2
  8. 2026VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inferencecited by 2
  9. 2026CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Modelscited by 2
  10. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  11. 2025Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Promptscited by 1
  12. 2024MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensingcited by 0
  13. 2025TLAC: Two-stage LMM Augmented CLIP for Zero-Shot Classificationcited by 0
  14. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learningcited by 0
  15. 2026CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysiscited by 0
  16. 2026Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Promptingcited by 0
  17. 2026From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition
  2. 2025SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models
  3. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Models
  4. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
  5. 2026AdaBoosting Text Prompts for Vision-Language Models
  6. 2023LaViP:Language-Grounded Visual Prompts
  7. 2023ECO: Ensembling Context Optimization for Vision-Language Models
  8. 2025AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs