Text2Model: Text-based Model Induction for Zero-shot Image Classification
Not on the map. Why?
Abstract (arXiv)
We address the challenge of building task-agnostic classifiers using only text descriptions, demonstrating a unified approach to image classification, 3D point cloud classification, and action recognition from scenes. Unlike approaches that learn a fixed representation of the output classes, we generate at inference time a model tailored to a query classification task. To generate task-based zero-shot classifiers, we train a hypernetwork that receives class descriptions and outputs a multi-class model. The hypernetwork is designed to be equivariant with respect to the set of descriptions and the classification layer, thus obeying the symmetries of the problem and improving generalization. Our approach generates non-linear classifiers, handles rich textual descriptions, and may be adapted to produce lightweight models efficient enough for on-device applications. We evaluate this approach in a series of zero-shot classification tasks, for image, point-cloud, and action recognition, using a range of text descriptions: From single words to rich descriptions. Our results demonstrate strong improvements over previous approaches, showing that zero-shot learning can be applied with little training data. Furthermore, we conduct an analysis with foundational vision and language models, demonstrating that they struggle to generalize when describing what attributes the class lacks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 48 of its 86 references that are in the map. First posted on arXiv 27 Oct 2022.
Builds on
48 of 86 references.
References · 48
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
- 2017Deep Setscited by 1,394
- 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
- 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
- 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
- 2017Feature Generating Networks for Zero-Shot Learningcited by 257
36 more
- 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
- 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
- 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
- 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
- 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 185
- 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
- 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
- 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
- 2017Equivariance Through Parameter-Sharingcited by 139
- 2020On Learning Sets of Symmetric Elementscited by 97
- 2017A Generative Adversarial Approach for Zero-Shot Learning from Noisy Textscited by 96
- 2021Contrastive Embedding for Generalized Zero-Shot Learningcited by 94
- 2020A causal view of compositional zero-shot recognitioncited by 78
- 2015Zero-Shot Learning via Semantic Similarity Embeddingcited by 74
- 2015Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptionscited by 64
- 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
- 2020Multimodal Prototypical Networks for Few-shot Learningcited by 48
- 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
- 2018Mapping Images to Scene Graphs with Permutation-Invariant Structured Predictioncited by 44
- 2018Transductive Unbiased Embedding for Zero-Shot Learningcited by 44
- 2017Context-aware Captions from Context-agnostic Supervisioncited by 43
- 2019Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspectivecited by 42
- 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
- 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
- 2018Preserving Semantic Relations for Zero-Shot Learningcited by 35
- 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
- 2016Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learningcited by 30
- 2020Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphscited by 30
- 2019Creativity Inspired Zero-Shot Learningcited by 27
- 2020ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarizationcited by 22
- 2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21
- 2022Example-based Hypernetworks for Out-of-Distribution Generalizationcited by 16
- 2017Semantically Consistent Regularization for Zero-Shot Recognitioncited by 14
- 2018Probabilistic AND-OR Attribute Grouping for Zero-Shot Learningcited by 6
- 2021Cross Knowledge-based Generative Zero-Shot Learning Approach with Taxonomy Regularizationcited by 5
- 2021Imaginative Walks: Generative Random Walk Deviation Loss for Improved Unseen Learning Representationcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.