mlsanityData to 7 Oct 2026

Rich Semantics Improve Few-shot Learning

arXiv 2104.12709 ↗First posted 26 Apr 2021cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Human learning benefits from multi-modal inputs that often appear as rich semantics (e.g., description of an object's attributes while learning about it). This enables us to learn generalizable concepts from very limited visual examples. However, current few-shot learning (FSL) methods use numerical class labels to denote object classes which do not provide rich semantic meanings about the learned concepts. In this work, we show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the FSL performance. Given a support set and queries, our main idea is to create a bottleneck visual feature (hybrid prototype) which is then used to generate language descriptions of the classes as an auxiliary task during training. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens that can encode intricate relationships between the two modalities. Forcing the prototypes to retain semantic information about class description acts as a regularizer on the visual features, improving their generalization to novel classes at inference. Furthermore, this strategy imposes a human prior on the learned representations, ensuring that the model is faithfully relating visual and semantic concepts, thereby improving model interpretability. Our experiments on four datasets and ablation studies show the benefit of effectively modeling rich semantics for FSL.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2019A Closer Look at Few-shot Classificationcited by 679
  12. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
20 more
  1. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  2. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  3. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  4. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  5. 2019Hyperbolic Image Embeddingscited by 219
  6. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
  7. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  8. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  9. 2019Boosting Few-Shot Visual Learning with Self-Supervisioncited by 140
  10. 2019Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learningcited by 121
  11. 2019Adaptive Cross-Modal Few-Shot Learningcited by 95
  12. 2019When Does Self-supervision Improve Few-shot Learning?cited by 78
  13. 2017Learning with Latent Languagecited by 75
  14. 2018Multi-level Semantic Feature Augmentation for One-shot Learningcited by 62
  15. 2021Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learningcited by 56
  16. 2020Self-supervised Knowledge Distillation for Few-shot Learningcited by 41
  17. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  18. 2020ALICE: Active Learning with Contrastive Natural Language Explanationscited by 36
  19. 2019Baby steps towards few-shot learning with multiple semanticscited by 29
  20. 2019Shaping Visual Representations with Language for Few-shot Classificationcited by 27

Built on by

Cited by year

Citing papers per year: 2022: 8; 2023: 3; 2024: 3; 2025: 1.

15 papers.

Sort:

By area:

  1. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  2. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  3. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 123
  4. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  5. 2024Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learningcited by 7
  6. 2022Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptionscited by 6
  7. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  8. 2024Cross-domain Multi-modal Few-shot Object Detection via Rich Textcited by 5
  9. 2022Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learningcited by 4
  10. 2022Visual-Semantic Contrastive Alignment for Few-Shot Image Classificationcited by 3
  11. 2022LAVA: Label-efficient Visual Learning and Adaptationcited by 1
  12. 2023Compositional Prototypical Networks for Few-Shot Classificationcited by 1
3 more
  1. 2022Few-Shot Classification of Skin Lesions from Dermoscopic Images by Meta-Learning Representative Embeddingscited by 0
  2. 2024Reading Is Believing: Revisiting Language Bottleneck Models for Image Classificationcited by 0
  3. 2025Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichmentcited by 0