# Shaping Visual Representations with Language for Few-shot Classification

arXiv: https://arxiv.org/abs/1911.02683 · HTML: https://mlsanity.com/paper/1911.02683/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2111.13579/) (2021)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [Fixing Model Bugs with Natural Language Patches](https://mlsanity.com/paper/2211.03318/) (2022)
- [On Guiding Visual Attention with Language Specification](https://mlsanity.com/paper/2202.08926/) (2022)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [Explaining Datasets in Words: Statistical Models with Natural Language Parameters](https://mlsanity.com/paper/2409.08466/) (2024)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis](https://mlsanity.com/paper/2405.14839/) (2024)
- [G^3: Geolocation via Guidebook Grounding](https://mlsanity.com/paper/2211.15521/) (2022)
- [Multimodality in Meta-Learning: A Comprehensive Survey](https://mlsanity.com/paper/2109.13576/) (2021)
- [Predictive Chemistry Augmented with Text Retrieval](https://mlsanity.com/paper/2312.04881/) (2023)
- [Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions](https://mlsanity.com/paper/2207.03133/) (2022)
- [Learning to Reason With Relational Abstractions](https://mlsanity.com/paper/2210.02615/) (2022)
- [TexRel: a Green Family of Datasets for Emergent Communications on Relations](https://mlsanity.com/paper/2105.12804/) (2021)
- [Visual-Semantic Contrastive Alignment for Few-Shot Image Classification](https://mlsanity.com/paper/2210.11000/) (2022)
- [Embedding Adaptation is Still Needed for Few-Shot Learning](https://mlsanity.com/paper/2104.07255/) (2021)
- [SemSup: Semantic Supervision for Simple and Scalable Zero-shot Generalization](https://mlsanity.com/paper/2202.13100/) (2022)
- [One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Features](https://mlsanity.com/paper/2404.19542/) (2024)
- [Semantics-driven Attentive Few-shot Learning over Clean and Noisy Samples](https://mlsanity.com/paper/2201.03043/) (2022)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification](https://mlsanity.com/paper/2406.15816/) (2024)

## Abstract (arXiv)

By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.