mlsanityData to 7 Oct 2026

Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition

arXiv 2212.04873 ↗First posted 9 Dec 2022cs.CV cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Current methods for few-shot action recognition mainly fall into the metric learning framework following ProtoNet, which demonstrates the importance of prototypes. Although they achieve relatively good performance, the effect of multimodal information is ignored, e.g. label texts. In this work, we propose a novel MultimOdal PRototype-ENhanced Network (MORN), which uses the semantic information of label texts as multimodal information to enhance prototypes. A CLIP visual encoder and a frozen CLIP text encoder are introduced to obtain features with good multimodal initialization. Then in the visual flow, visual prototypes are computed by a visual prototype-computed module. In the text flow, a semantic-enhanced (SE) module and an inflating operation are used to obtain text prototypes. The final multimodal prototypes are then computed by a multimodal prototype-enhanced (MPE) module. Besides, we define a PRototype SImilarity DiffErence (PRIDE) to evaluate the quality of prototypes, which is used to verify our improvement on the prototype level and effectiveness of MORN. We conduct extensive experiments on four popular few-shot action recognition datasets: HMDB51, UCF101, Kinetics and SSv2, and MORN achieves state-of-the-art results. When plugging PRIDE into the training stage, the performance can be further improved.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

50 of 67 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
38 more
  1. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  3. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2016Learning to learn by gradient descent by gradient descentcited by 616
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  8. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  9. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  10. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  11. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  12. 2018How to train your MAMLcited by 306
  13. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  14. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  15. 2019Cross Attention Network for Few-shot Classificationcited by 218
  16. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  17. 2018Task-Agnostic Meta-Learning for Few-shot Learningcited by 151
  18. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  19. 2019Few-Shot Video Classification via Temporal Alignmentcited by 124
  20. 2019Edge-labeling Graph Neural Network for Few-shot Learningcited by 114
  21. 2019Finding Task-Relevant Features for Few-Shot Learning by Category Traversalcited by 101
  22. 2017Learning to Compose Domain-Specific Transformations for Data Augmentationcited by 91
  23. 2019TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learningcited by 90
  24. 2021Temporal-Relational CrossTransformers for Few-Shot Action Recognitioncited by 83
  25. 2020DPGN: Distribution Propagation Graph Network for Few-shot Learningcited by 79
  26. 2019TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognitioncited by 71
  27. 2018Multi-level Semantic Feature Augmentation for One-shot Learningcited by 62
  28. 2021Spatio-temporal Relation Modeling for Few-shot Action Recognitioncited by 52
  29. 2020Prototype Completion with Primitive Knowledge for Few-Shot Learningcited by 50
  30. 2020Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognitioncited by 50
  31. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 49
  32. 2020Multimodal Prototypical Networks for Few-shot Learningcited by 48
  33. 2019ProtoGAN: Towards Few Shot Learning for Action Recognitioncited by 41
  34. 2019An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learningcited by 38
  35. 2022Compound Prototype Matching for Few-shot Action Recognitioncited by 35
  36. 2021Few-shot Action Recognition with Prototype-centered Attentive Learningcited by 28
  37. 2021Learning Implicit Temporal Alignment for Few-shot Video Classificationcited by 27
  38. 2018Self Paced Adversarial Training for Multimodal Few-shot Learningcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 4; 2025: 4.

13 papers.

Sort:

By area:

  1. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  2. 2023Boosting Few-shot Action Recognition with Graph-guided Hybrid Matchingcited by 24
  3. 2023Global and Local Semantic Completion Learning for Vision-Language Pre-trainingcited by 12
  4. 2024A Comprehensive Review of Few-shot Action Recognitioncited by 11
  5. 2024Trajectory-aligned Space-time Tokens for Few-shot Action Recognitioncited by 10
  6. 2023Semantic-aware Video Representation for Few-shot Action Recognitioncited by 7
  7. 2025Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognitioncited by 7
  8. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  9. 2023Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CPM2C)cited by 4
  10. 2024Multi-view Distillation based on Multi-modal Fusion for Few-shot Action Recognition(CLIP-M2DF)cited by 2
  11. 2025Beyond Label Semantics: Language-Guided Action Anatomy for Few-shot Action Recognitioncited by 2
  12. 2024Fully Aligned Network for Referring Image Segmentationcited by 0
1 more
  1. 2025Distinguishing Visually Similar Actions: Prompt-Guided Semantic Prototype Modulation for Few-Shot Action Recognitioncited by 0