# Multimodal Prototypical Networks for Few-shot Learning

arXiv: https://arxiv.org/abs/2011.08899 · HTML: https://mlsanity.com/paper/2011.08899/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 30 of 48)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Hyperbolic Busemann Learning with Ideal Prototypes](https://mlsanity.com/paper/2106.14472/) (2021)
- [CLIP-guided Prototype Modulating for Few-shot Action Recognition](https://mlsanity.com/paper/2303.02982/) (2023)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification](https://mlsanity.com/paper/2211.16191/) (2022)
- [Meta-learning approaches for few-shot learning: A survey of recent advances](https://mlsanity.com/paper/2303.07502/) (2023)
- [Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations](https://mlsanity.com/paper/2112.02290/) (2021)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [A Closer Look at Prototype Classifier for Few-shot Image Classification](https://mlsanity.com/paper/2110.05076/) (2021)
- [Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition](https://mlsanity.com/paper/2212.04873/) (2022)
- [Multimodality in Meta-Learning: A Comprehensive Survey](https://mlsanity.com/paper/2109.13576/) (2021)
- [Towards Effective Visual Representations for Partial-Label Learning](https://mlsanity.com/paper/2305.06080/) (2023)
- [HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities](https://mlsanity.com/paper/2105.07889/) (2021)
- [Episodic Multi-Task Learning with Heterogeneous Neural Processes](https://mlsanity.com/paper/2310.18713/) (2023)
- [Visual Hindsight Self-Imitation Learning for Interactive Navigation](https://mlsanity.com/paper/2312.03446/) (2023)
- [Hierarchical Explanations for Video Action Recognition](https://mlsanity.com/paper/2301.00436/) (2023)
- [The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagery](https://mlsanity.com/paper/2304.02016/) (2023)
- [ProtoEM: A Prototype-Enhanced Matching Framework for Event Relation Extraction](https://mlsanity.com/paper/2309.12892/) (2023)
- [Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning](https://mlsanity.com/paper/2106.14467/) (2021)
- [Kernel Relative-prototype Spectral Filtering for Few-shot Learning](https://mlsanity.com/paper/2207.11685/) (2022)
- [Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CPM2C)](https://mlsanity.com/paper/2312.01083/) (2023)
- [DMSD-CDFSAR: Distillation from Mixed-Source Domain for Cross-Domain Few-shot Action Recognition](https://mlsanity.com/paper/2407.05657/) (2024)
- [Few-shot Partial Multi-view Learning](https://mlsanity.com/paper/2105.02046/) (2021)
- [Predicting emergent linguistic compositions through time: Syntactic frame extension via multimodal chaining](https://mlsanity.com/paper/2109.04652/) (2021)
- [Robust Text Classification: Analyzing Prototype-Based Networks](https://mlsanity.com/paper/2311.06647/) (2023)
- [Shaping Visual Representations with Attributes for Few-Shot Recognition](https://mlsanity.com/paper/2112.06398/) (2021)
- [Task-Specific Alignment and Multiple Level Transformer for Few-Shot Action Recognition](https://mlsanity.com/paper/2307.01985/) (2023)
- [On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval](https://mlsanity.com/paper/2311.00693/) (2023)
- [Prototype-Aware Heterogeneous Task for Point Cloud Completion](https://mlsanity.com/paper/2209.01733/) (2022)

## Abstract (arXiv)

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.