# Rich Semantics Improve Few-shot Learning

arXiv: https://arxiv.org/abs/2104.12709 · HTML: https://mlsanity.com/paper/2104.12709/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Boosting Few-Shot Visual Learning with Self-Supervision](https://mlsanity.com/paper/1906.05186/) (2019)
- [Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning](https://mlsanity.com/paper/1903.12290/) (2019)
- [Adaptive Cross-Modal Few-Shot Learning](https://mlsanity.com/paper/1902.07104/) (2019)
- [When Does Self-supervision Improve Few-shot Learning?](https://mlsanity.com/paper/1910.03560/) (2019)
- [Learning with Latent Language](https://mlsanity.com/paper/1711.00482/) (2017)
- [Multi-level Semantic Feature Augmentation for One-shot Learning](https://mlsanity.com/paper/1804.05298/) (2018)
- [Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning](https://mlsanity.com/paper/2103.01315/) (2021)
- [Self-supervised Knowledge Distillation for Few-shot Learning](https://mlsanity.com/paper/2006.09785/) (2020)
- [ShapeWorld - A new test methodology for multimodal language understanding](https://mlsanity.com/paper/1704.04517/) (2017)
- [ALICE: Active Learning with Contrastive Natural Language Explanations](https://mlsanity.com/paper/2009.10259/) (2020)

## Built on by (most cited first, 15 of 15)

- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2403.04492/) (2024)
- [Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions](https://mlsanity.com/paper/2207.03133/) (2022)
- [BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoning](https://mlsanity.com/paper/2309.01256/) (2023)
- [Cross-domain Multi-modal Few-shot Object Detection via Rich Text](https://mlsanity.com/paper/2403.16188/) (2024)
- [Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning](https://mlsanity.com/paper/2201.05151/) (2022)
- [Visual-Semantic Contrastive Alignment for Few-Shot Image Classification](https://mlsanity.com/paper/2210.11000/) (2022)
- [LAVA: Label-efficient Visual Learning and Adaptation](https://mlsanity.com/paper/2210.10317/) (2022)
- [Compositional Prototypical Networks for Few-Shot Classification](https://mlsanity.com/paper/2306.06584/) (2023)
- [Few-Shot Classification of Skin Lesions from Dermoscopic Images by Meta-Learning Representative Embeddings](https://mlsanity.com/paper/2210.16954/) (2022)
- [Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification](https://mlsanity.com/paper/2406.15816/) (2024)
- [Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment](https://mlsanity.com/paper/2502.16469/) (2025)

## Abstract (arXiv)

Human learning benefits from multi-modal inputs that often appear as rich semantics (e.g., description of an object's attributes while learning about it). This enables us to learn generalizable concepts from very limited visual examples. However, current few-shot learning (FSL) methods use numerical class labels to denote object classes which do not provide rich semantic meanings about the learned concepts. In this work, we show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the FSL performance. Given a support set and queries, our main idea is to create a bottleneck visual feature (hybrid prototype) which is then used to generate language descriptions of the classes as an auxiliary task during training. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens that can encode intricate relationships between the two modalities. Forcing the prototypes to retain semantic information about class description acts as a regularizer on the visual features, improving their generalization to novel classes at inference. Furthermore, this strategy imposes a human prior on the learned representations, ensuring that the model is faithfully relating visual and semantic concepts, thereby improving model interpretability. Our experiments on four datasets and ablation studies show the benefit of effectively modeling rich semantics for FSL.