# Adaptive Cross-Modal Few-Shot Learning

arXiv: https://arxiv.org/abs/1902.07104 · HTML: https://mlsanity.com/paper/1902.07104/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2019.

## Built on by (most cited first, 30 of 95)

- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias](https://mlsanity.com/paper/2101.07296/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Learning from Few Examples: A Summary of Approaches to Few-Shot Learning](https://mlsanity.com/paper/2203.04291/) (2022)
- [Shallow Bayesian Meta Learning for Real-World Few-Shot Recognition](https://mlsanity.com/paper/2101.02833/) (2021)
- [CLIP-guided Prototype Modulating for Few-shot Action Recognition](https://mlsanity.com/paper/2303.02982/) (2023)
- [Few-Shot Object Detection via Association and DIscrimination](https://mlsanity.com/paper/2111.11656/) (2021)
- [Generating Representative Samples for Few-Shot Classification](https://mlsanity.com/paper/2205.02918/) (2022)
- [SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification](https://mlsanity.com/paper/2211.16191/) (2022)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Few-Shot Classification with Contrastive Learning](https://mlsanity.com/paper/2209.08224/) (2022)
- [Semantic Prompt for Few-Shot Image Recognition](https://mlsanity.com/paper/2303.14123/) (2023)
- [Reliable Few-shot Learning under Dual Noises](https://mlsanity.com/paper/2506.16330/) (2025)
- [Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling](https://mlsanity.com/paper/2403.10071/) (2024)
- [Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning](https://mlsanity.com/paper/2101.09499/) (2021)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [SEGA: Semantic Guided Attention on Visual Prototype for Few-Shot Learning](https://mlsanity.com/paper/2111.04316/) (2021)
- [Inferring Prototypes for Multi-Label Few-Shot Image Classification with Word Vector Guided Attention](https://mlsanity.com/paper/2112.01037/) (2021)
- [Simple Semantic-Aided Few-Shot Learning](https://mlsanity.com/paper/2311.18649/) (2023)
- [DETA: Denoised Task Adaptation for Few-Shot Learning](https://mlsanity.com/paper/2303.06315/) (2023)
- [Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning](https://mlsanity.com/paper/2105.10195/) (2021)
- [FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning](https://mlsanity.com/paper/2310.15105/) (2023)
- [Audio-to-Image Cross-Modal Generation](https://mlsanity.com/paper/2109.13354/) (2021)
- [Multimodality in Meta-Learning: A Comprehensive Survey](https://mlsanity.com/paper/2109.13576/) (2021)
- [Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting](https://mlsanity.com/paper/2204.07841/) (2022)
- [Modelling Commonsense Properties using Pre-Trained Bi-Encoders](https://mlsanity.com/paper/2210.02771/) (2022)

## Abstract (arXiv)

Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, visual features might be richer and more discriminative than text ones. While for others, the inverse might be true. Moreover, when the support from visual information is limited in image classification, semantic representations (learned from unsupervised text corpora) can provide strong prior knowledge and context to help learning. Based on these two intuitions, we propose a mechanism that can adaptively combine information from both modalities according to new image categories to be learned. Through a series of experiments, we show that by this adaptive combination of the two modalities, our model outperforms current uni-modality few-shot learning methods and modality-alignment methods by a large margin on all benchmarks and few-shot scenarios tested. Experiments also show that our model can effectively adjust its focus on the two modalities. The improvement in performance is particularly large when the number of shots is very small.