# Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions

arXiv: https://arxiv.org/abs/2207.03133 · HTML: https://mlsanity.com/paper/2207.03133/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Adaptive Cross-Modal Few-Shot Learning](https://mlsanity.com/paper/1902.07104/) (2019)
- [Grounding Visual Explanations](https://mlsanity.com/paper/1807.09685/) (2018)
- [Learning with Latent Language](https://mlsanity.com/paper/1711.00482/) (2017)
- [Fine-graind Image Classification via Combining Vision and Language](https://mlsanity.com/paper/1704.02792/) (2017)
- [ALICE: Active Learning with Contrastive Natural Language Explanations](https://mlsanity.com/paper/2009.10259/) (2020)
- [Shaping Visual Representations with Language for Few-shot Classification](https://mlsanity.com/paper/1911.02683/) (2019)
- [Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions](https://mlsanity.com/paper/1801.09041/) (2018)
- [Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner](https://mlsanity.com/paper/1705.00930/) (2017)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classification](https://mlsanity.com/paper/2106.11486/) (2021)

## Built on by (most cited first, 6 of 6)

- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Explanation Bottleneck Models](https://mlsanity.com/paper/2409.17663/) (2024)
- [Coreset Selection via LLM-based Concept Bottlenecks](https://mlsanity.com/paper/2502.16733/) (2025)
- [LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classification](https://mlsanity.com/paper/2405.18672/) (2024)
- [Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification](https://mlsanity.com/paper/2406.15816/) (2024)

## Abstract (arXiv)

Humans can obtain the knowledge of novel visual concepts from language descriptions, and we thus use the few-shot image classification task to investigate whether a machine learning model can have this capability. Our proposed model, LIDE (Learning from Image and DEscription), has a text decoder to generate the descriptions and a text encoder to obtain the text representations of machine- or user-generated descriptions. We confirmed that LIDE with machine-generated descriptions outperformed baseline models. Moreover, the performance was improved further with high-quality user-generated descriptions. The generated descriptions can be viewed as the explanations of the model's predictions, and we observed that such explanations were consistent with prediction results. We also investigated why the language description improved the few-shot image classification performance by comparing the image representations and the text representations in the feature spaces.