mlsanityData to 7 Oct 2026

Will Multi-modal Data Improves Few-shot Learning?

arXiv 2107.11853 ↗First posted 25 Jul 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Most few-shot learning models utilize only one modality of data. We would like to investigate qualitatively and quantitatively how much will the model improve if we add an extra modality (i.e. text description of the image), and how it affects the learning procedure. To achieve this goal, we propose four types of fusion method to combine the image feature and text feature. To verify the effectiveness of improvement, we test the fusion methods with two classical few-shot learning models - ProtoNet and MAML, with image feature extractors such as ConvNet and ResNet12. The attention-based fusion method works best, which improves the classification accuracy by a large margin around 30% comparing to the baseline result.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 25 Jul 2021.

Builds on

30 of 40 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2016Matching Networks for One Shot Learningcited by 2,246
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2019A Closer Look at Few-shot Classificationcited by 679
18 more
  1. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  2. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  3. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  4. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  5. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  6. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 208
  7. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  8. 2015Compact Bilinear Poolingcited by 97
  9. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  10. 2020DPGN: Distribution Propagation Graph Network for Few-shot Learningcited by 79
  11. 2015Where To Look: Focus Regions for Visual Question Answeringcited by 64
  12. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  13. 2020DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learningcited by 49
  14. 2020Multimodal Prototypical Networks for Few-shot Learningcited by 48
  15. 2018Few-Shot Learning with Metric-Agnostic Conditional Embeddingscited by 40
  16. 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29
  17. 2015Simple Baseline for Visual Question Answeringcited by 24
  18. 2020Direct multimodal few-shot learning of speech and imagescited by 6

Built on by

Not cited yet.