# Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition

arXiv: https://arxiv.org/abs/2212.04873 · HTML: https://mlsanity.com/paper/2212.04873/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 67 references that are in the map.
- First posted on arXiv 9 Dec 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Task-Agnostic Meta-Learning for Few-shot Learning](https://mlsanity.com/paper/1805.07722/) (2018)

## Built on by (most cited first, 13 of 13)

- [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023)
- [Boosting Few-shot Action Recognition with Graph-guided Hybrid Matching](https://mlsanity.com/paper/2308.09346/) (2023)
- [Global and Local Semantic Completion Learning for Vision-Language Pre-training](https://mlsanity.com/paper/2306.07096/) (2023)
- [A Comprehensive Review of Few-shot Action Recognition](https://mlsanity.com/paper/2407.14744/) (2024)
- [Trajectory-aligned Space-time Tokens for Few-shot Action Recognition](https://mlsanity.com/paper/2407.18249/) (2024)
- [Semantic-aware Video Representation for Few-shot Action Recognition](https://mlsanity.com/paper/2311.06218/) (2023)
- [Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition](https://mlsanity.com/paper/2508.03695/) (2025)
- [Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2508.01225/) (2025)
- [Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CPM2C)](https://mlsanity.com/paper/2312.01083/) (2023)
- [Multi-view Distillation based on Multi-modal Fusion for Few-shot Action Recognition(CLIP-M2DF)](https://mlsanity.com/paper/2401.08345/) (2024)
- [Beyond Label Semantics: Language-Guided Action Anatomy for Few-shot Action Recognition](https://mlsanity.com/paper/2507.16287/) (2025)
- [Fully Aligned Network for Referring Image Segmentation](https://mlsanity.com/paper/2409.19569/) (2024)
- [Distinguishing Visually Similar Actions: Prompt-Guided Semantic Prototype Modulation for Few-Shot Action Recognition](https://mlsanity.com/paper/2512.19036/) (2025)

## Abstract (arXiv)

Current methods for few-shot action recognition mainly fall into the metric learning framework following ProtoNet, which demonstrates the importance of prototypes. Although they achieve relatively good performance, the effect of multimodal information is ignored, e.g. label texts. In this work, we propose a novel MultimOdal PRototype-ENhanced Network (MORN), which uses the semantic information of label texts as multimodal information to enhance prototypes. A CLIP visual encoder and a frozen CLIP text encoder are introduced to obtain features with good multimodal initialization. Then in the visual flow, visual prototypes are computed by a visual prototype-computed module. In the text flow, a semantic-enhanced (SE) module and an inflating operation are used to obtain text prototypes. The final multimodal prototypes are then computed by a multimodal prototype-enhanced (MPE) module. Besides, we define a PRototype SImilarity DiffErence (PRIDE) to evaluate the quality of prototypes, which is used to verify our improvement on the prototype level and effectiveness of MORN. We conduct extensive experiments on four popular few-shot action recognition datasets: HMDB51, UCF101, Kinetics and SSv2, and MORN achieves state-of-the-art results. When plugging PRIDE into the training stage, the performance can be further improved.