# Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision

arXiv: https://arxiv.org/abs/1709.01148 · HTML: https://mlsanity.com/paper/1709.01148/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2017.

## Built on by (most cited first, 30 of 46)

- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [PromptDet: Towards Open-vocabulary Detection using Uncurated Images](https://mlsanity.com/paper/2203.16513/) (2022)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [K-LITE: Learning Transferable Visual Models with External Knowledge](https://mlsanity.com/paper/2204.09222/) (2022)
- [Open-Set Image Tagging with Multi-Grained Text Supervision](https://mlsanity.com/paper/2310.15200/) (2023)
- [I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2212.02291/) (2022)
- [Multi-Modal Classifiers for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.05493/) (2023)
- [I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification](https://mlsanity.com/paper/2209.10304/) (2022)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions](https://mlsanity.com/paper/2103.09669/) (2021)
- [Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classification](https://mlsanity.com/paper/2203.01386/) (2022)
- [Prompting Scientific Names for Zero-Shot Species Recognition](https://mlsanity.com/paper/2310.09929/) (2023)
- [Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images](https://mlsanity.com/paper/2311.01064/) (2023)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)
- [Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2101.03292/) (2021)
- [The Curious Layperson: Fine-Grained Image Recognition without Expert Labels](https://mlsanity.com/paper/2111.03651/) (2021)
- [PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck](https://mlsanity.com/paper/2403.05297/) (2024)
- [CIZSL++: Creativity Inspired Generative Zero-Shot Learning](https://mlsanity.com/paper/2101.00173/) (2021)
- [Cross Knowledge-based Generative Zero-Shot Learning Approach with Taxonomy Regularization](https://mlsanity.com/paper/2101.09892/) (2021)
- [Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs](https://mlsanity.com/paper/2107.06563/) (2021)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [An Entropy-guided Reinforced Partial Convolutional Network for Zero-Shot Learning](https://mlsanity.com/paper/2111.02139/) (2021)
- [Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learning](https://mlsanity.com/paper/2407.15613/) (2024)
- [An Iterative Co-Training Transductive Framework for Zero Shot Learning](https://mlsanity.com/paper/2203.16041/) (2022)
- [Imaginative Walks: Generative Random Walk Deviation Loss for Improved Unseen Learning Representation](https://mlsanity.com/paper/2104.09757/) (2021)
- [Zero-shot Learning with Class Description Regularization](https://mlsanity.com/paper/2106.16108/) (2021)
- [Continual Zero-Shot Learning through Semantically Guided Generative Random Walks](https://mlsanity.com/paper/2308.12366/) (2023)

## Abstract (arXiv)

In this paper, we study learning visual classifiers from unstructured text descriptions at part precision with no training images. We propose a learning framework that is able to connect text terms to its relevant parts and suppress connections to non-visual text terms without any part-text annotations. For instance, this learning process enables terms like "beak" to be sparsely linked to the visual representation of parts like head, while reduces the effect of non-visual terms like "migrate" on classifier prediction. Images are encoded by a part-based CNN that detect bird parts and learn part-specific representation. Part-based visual classifiers are predicted from text descriptions of unseen visual classifiers to facilitate classification without training images (also known as zero-shot recognition). We performed our experiments on CUBirds 2011 dataset and improves the state-of-the-art text-based zero-shot recognition results from 34.7% to 43.6%. We also created large scale benchmarks on North American Bird Images augmented with text descriptions, where we also show that our approach outperforms existing methods. Our code, data, and models are publically available.