# Text2Model: Text-based Model Induction for Zero-shot Image Classification

arXiv: https://arxiv.org/abs/2210.15182 · HTML: https://mlsanity.com/paper/2210.15182/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 86 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data](https://mlsanity.com/paper/2002.12880/) (2020)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [On Learning Sets of Symmetric Elements](https://mlsanity.com/paper/2002.08599/) (2020)
- [A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts](https://mlsanity.com/paper/1712.01381/) (2017)
- [Contrastive Embedding for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2103.16173/) (2021)
- [A causal view of compositional zero-shot recognition](https://mlsanity.com/paper/2006.14610/) (2020)
- [Zero-Shot Learning via Semantic Similarity Embedding](https://mlsanity.com/paper/1509.04767/) (2015)
- [Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions](https://mlsanity.com/paper/1506.00511/) (2015)
- [Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2108.06230/) (2021)
- [Multimodal Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/2011.08899/) (2020)
- [Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision](https://mlsanity.com/paper/1709.01148/) (2017)

## Built on by (most cited first, 1 of 1)

- [WeightCLIP: Aligning Datasets and Models for Weight Space Learning](https://mlsanity.com/paper/2607.03551/) (2026)

## Abstract (arXiv)

We address the challenge of building task-agnostic classifiers using only text descriptions, demonstrating a unified approach to image classification, 3D point cloud classification, and action recognition from scenes. Unlike approaches that learn a fixed representation of the output classes, we generate at inference time a model tailored to a query classification task. To generate task-based zero-shot classifiers, we train a hypernetwork that receives class descriptions and outputs a multi-class model. The hypernetwork is designed to be equivariant with respect to the set of descriptions and the classification layer, thus obeying the symmetries of the problem and improving generalization. Our approach generates non-linear classifiers, handles rich textual descriptions, and may be adapted to produce lightweight models efficient enough for on-device applications. We evaluate this approach in a series of zero-shot classification tasks, for image, point-cloud, and action recognition, using a range of text descriptions: From single words to rich descriptions. Our results demonstrate strong improvements over previous approaches, showing that zero-shot learning can be applied with little training data. Furthermore, we conduct an analysis with foundational vision and language models, demonstrating that they struggle to generalize when describing what attributes the class lacks.