# ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization

arXiv: https://arxiv.org/abs/2010.03276 · HTML: https://mlsanity.com/paper/2010.03276/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Open-Set Image Tagging with Multi-Grained Text Supervision](https://mlsanity.com/paper/2310.15200/) (2023)
- [Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images](https://mlsanity.com/paper/2311.01064/) (2023)
- [Zero-Shot Aerial Object Detection with Visual Description Regularization](https://mlsanity.com/paper/2402.18233/) (2024)
- [PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck](https://mlsanity.com/paper/2403.05297/) (2024)
- [Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP](https://mlsanity.com/paper/2412.11375/) (2024)
- [BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision](https://mlsanity.com/paper/2309.12056/) (2023)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [HeGeL: A Novel Dataset for Geo-Location from Hebrew Text](https://mlsanity.com/paper/2307.00509/) (2023)
- [Where Do We Go from Here? Multi-scale Allocentric Relational Inference from Natural Spatial Descriptions](https://mlsanity.com/paper/2402.16364/) (2024)
- [EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipedia](https://mlsanity.com/paper/2504.19742/) (2025)
- [Towards Automated Knowledge Integration From Human-Interpretable Representations](https://mlsanity.com/paper/2402.16105/) (2024)
- [Text2Model: Text-based Model Induction for Zero-shot Image Classification](https://mlsanity.com/paper/2210.15182/) (2022)
- [Into the Unknown: Generating Geospatial Descriptions for New Environments](https://mlsanity.com/paper/2406.19967/) (2024)
- [Multi-Modal Fusion by Meta-Initialization](https://mlsanity.com/paper/2210.04843/) (2022)
- [Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions](https://mlsanity.com/paper/2303.04053/) (2023)
- [Apollo: Zero-shot MultiModal Reasoning with Multiple Experts](https://mlsanity.com/paper/2310.18369/) (2023)
- [Data Alignment for Zero-Shot Concept Generation in Dermatology AI](https://mlsanity.com/paper/2404.13043/) (2024)
- [Multi-method Integration with Confidence-based Weighting for Zero-shot Image Classification](https://mlsanity.com/paper/2405.02155/) (2024)

## Abstract (arXiv)

We study the problem of recognizing visual entities from the textual descriptions of their classes. Specifically, given birds' images with free-text descriptions of their species, we learn to classify images of previously-unseen species based on specie descriptions. This setup has been studied in the vision community under the name zero-shot learning from text, focusing on learning to transfer knowledge about visual aspects of birds from seen classes to previously-unseen ones. Here, we suggest focusing on the textual description and distilling from the description the most relevant information to effectively match visual features to the parts of the text that discuss them. Specifically, (1) we propose to leverage the similarity between species, reflected in the similarity between text descriptions of the species. (2) we derive visual summaries of the texts, i.e., extractive summaries that focus on the visual features that tend to be reflected in images. We propose a simple attention-based model augmented with the similarity and visual summaries components. Our empirical results consistently and significantly outperform the state-of-the-art on the largest benchmarks for text-based zero-shot learning, illustrating the critical importance of texts for zero-shot image-recognition.