# I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification

arXiv: https://arxiv.org/abs/2212.02291 · HTML: https://mlsanity.com/paper/2212.02291/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 66 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild](https://mlsanity.com/paper/1605.04253/) (2016)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Generalized Zero-Shot Learning via Synthesized Examples](https://mlsanity.com/paper/1712.03878/) (2017)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)

## Built on by (most cited first, 30 of 54)

- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [SILC: Improving Vision Language Pretraining with Self-Distillation](https://mlsanity.com/paper/2310.13355/) (2023)
- [Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions](https://mlsanity.com/paper/2401.02460/) (2024)
- [Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts](https://mlsanity.com/paper/2307.11661/) (2023)
- [Introducing Language Guidance in Prompt-based Continual Learning](https://mlsanity.com/paper/2308.15827/) (2023)
- [UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All](https://mlsanity.com/paper/2403.12532/) (2024)
- [SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance](https://mlsanity.com/paper/2311.16241/) (2023)
- [Learning to Prompt with Text Only Supervision for Vision-Language Models](https://mlsanity.com/paper/2401.02418/) (2024)
- [Agent3D-Zero: An Agent for Zero-shot 3D Understanding](https://mlsanity.com/paper/2403.11835/) (2024)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeek](https://mlsanity.com/paper/2505.17702/) (2025)
- [Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2404.07713/) (2024)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [Image-free Classifier Injection for Zero-Shot Classification](https://mlsanity.com/paper/2308.10599/) (2023)
- [Open-Vocabulary Calibration for Fine-tuned CLIP](https://mlsanity.com/paper/2402.04655/) (2024)
- [ExpLLM: Towards Chain of Thought for Facial Expression Recognition](https://mlsanity.com/paper/2409.02828/) (2024)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection](https://mlsanity.com/paper/2311.01025/) (2023)
- [Context-Enhanced Video Moment Retrieval with Large Language Models](https://mlsanity.com/paper/2405.12540/) (2024)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)
- [Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge](https://mlsanity.com/paper/2310.15066/) (2023)
- [A Survey on Open-Set Image Recognition](https://mlsanity.com/paper/2312.15571/) (2023)
- [LLMBind: A Unified Modality-Task Integration Framework](https://mlsanity.com/paper/2402.14891/) (2024)
- [AquaticCLIP: A Vision-Language Foundation Model for Underwater Scene Analysis](https://mlsanity.com/paper/2502.01785/) (2025)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder](https://mlsanity.com/paper/2505.11882/) (2025)
- [SpatialMosaic: A Multiview VLM Dataset for Partial Visibility](https://mlsanity.com/paper/2512.23365/) (2025)
- [Open-Pose 3D Zero-Shot Learning: Benchmark and Challenges](https://mlsanity.com/paper/2312.07039/) (2023)
- [ECOR: Explainable CLIP for Object Recognition](https://mlsanity.com/paper/2404.12839/) (2024)
- [Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model](https://mlsanity.com/paper/2407.04489/) (2024)

## Abstract (arXiv)

Recent works have shown that unstructured text (documents) from online sources can serve as useful auxiliary information for zero-shot image classification. However, these methods require access to a high-quality source like Wikipedia and are limited to a single source of information. Large Language Models (LLM) trained on web-scale text show impressive abilities to repurpose their learned knowledge for a multitude of tasks. In this work, we provide a novel perspective on using an LLM to provide text supervision for a zero-shot image classification model. The LLM is provided with a few text descriptions from different annotators as examples. The LLM is conditioned on these examples to generate multiple text descriptions for each class(referred to as views). Our proposed model, I2MVFormer, learns multi-view semantic embeddings for zero-shot image classification with these class views. We show that each text view of a class provides complementary information allowing a model to learn a highly discriminative class embedding. Moreover, we show that I2MVFormer is better at consuming the multi-view text supervision from LLM compared to baseline models. I2MVFormer establishes a new state-of-the-art on three public benchmark datasets for zero-shot image classification with unsupervised semantic embeddings.